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’Y  Improving  Cost  Estimating  in  the  Department  of  Defense^ 

Milton  A.  hiargolis 

Improving  cost  estimates  for  military  systems  continues  to  receive  high  priori¬ 
ty  within  the  Department  of  Defense.  Taking  the  lead  in  this  effort «  the  OSD 
Cost  Analysis  Improvement  Group  (CAIC).  The  author  discussesjhe  CAIC  and 
its  role  in  reviewing  service  cost  estimates  and  fostering  defense-wide  im¬ 
provements  in.  military  cost  analysis  capabilities. 

JQ '^Estimating  in  the  '80s — Special  Section  y 

The  Fifteenth  Annual  Cost  Analysis  Symppsfum  was  held  in  October  1980. 
Part  of  that  symposium  was  devoted  to  a  ptdtel  discussion  revolving  around  the 
theme  "Estimating  in  the  '80s. "  This  special  section  includes  some  of  the  presenta¬ 
tions  by  panel  members  and  a  portion  of  the  transcript  from  the  question-and- 
answer  period. 

48  Alternative  Techniques  for  Use  in  Parametric  Cost  Analysis  .* 

W.  Eugene  Wfller 
Thomas  J.  Dlvyer 

In  an  effort  to  eliminate  some  of  the  difficulties  in  developing  a  cost  estimating 
model  or  a  point  estimate  for  military  systems,  the  authors  present  (1)  some  alter¬ 
natives  to  the  conventional  statistical  methods  of  developing  cost  estimating 
models,  and  Q)  a  means  for  normalizing  cost  estimating  models  so  that  the 
models  stay  relevant  in  the  face  of  rapidly  advancing  technology. 

'V'' 

60  MX  Cost  Analysis  »  Major  Donald  E.  Crawford.  USAF 

The  magnitude  of  ffte  MX  missile  program  and  its  political  implications, 
makes  even  more  critical  the  development  of  valid  cost  esi  lates.  The  author 
describes  the  computerized  cost  model  designed  to  develop  those  tjlltnale^  I 
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63  Three  Views  of  the  Impact  of  Production  Rate  Changes^  • 

( Redistributing  Fixed  Overhead  Costs^ 

Cj)mmander  Steve  /.  Balut.  USN 

Cost  analysts  in  OSD  (PA6E)  involved  in  developing  alternatives  to  pro¬ 
posed  system  procurement  quantities  and  rates  must  first  derive  new  average  unit 
prices  for  each  lot  consistent  with  new  lot  quantities,  and  then  adjust  prices  to 
reflect  the  attendant  redistribution  of  fixed  overhead.  The  author  focuses  his 
discussion  on  the  second  part  of  this  calculation,  presenting  a  model  that  may  be 
in  making  such  "rate  adjustments. " 

Effect  of  Production  Rate  on  Weapon  System  Cost^ . 

Dr.  Charl^  H.  Smith 

Adjustments  to  production  rates  during  the  planning  and  production  of  a  new 
system  have  a  direct  impact  on  unit  cost.  DeterBUHingthe  extent  of  this  impact  is 
the  job  of  the  cost  analyst.  The  author  addresses  the  current  state  of  production 
rate  research  and  proposes  a  technique  applicable  to  future  cost  analysis. 

84rnil  A  Model  for  Examining  the  Cost  Implications  of 

Production  Ratey  John  C.  Bemis 

One  of  the  varial^s  affecting  unit  cost  of  a  system  or  item  is  production  rate. 
The  author  presents  a  model  that  relates  production  rate,  unit  cost,  and 
cumulative  quantity.  According  to  the  author,  the  model  can  be  used  to  assist  in 
the  selection  of  the  most  economical  production  rate  for  a  wide  variety  of  com¬ 
plex  systems. 

V 

95  The  VAMOSC  Connection:  Improving  Operating  and 

Support  Costing  ^  \  Alvin  M.  Frager 

Many  OOD  initiatitvs  are  aimed  at  reducing  the  cost  of  new  weapon  systems.  This 
article  describes  the  visibility  and  management  of  support  cost  (VAMOSC)  in¬ 
itiative  to  identify  the  operating  and  support  (Of'S)  costs  of  existing  hardware. 
Mr.  Frager  says  the  application  of  VAMOSC  data  has  already  been  used  to 
estimate  O&S  costs  of  new  systems,  and  explains  how  it  will  provide  more  effec¬ 
tive  decision  criteria  to  program  managers. 


105  Growth  Being  Reinforced^!  Noreen  S.  Bryan 

\  Dr.  Rolf  Clark 

Investigations  into  cost  growth  in  the  development  of  military  systems  have 
reveaitd  such  causes  as  inflation,  technical  and  protfuction  rate  changes,  overop¬ 
timism  and  "buy-ins.  "  The  authors  suggest  that  another  cause  could  be  current 
cost  estimating  practices. 


119  Meeting  the  Challenge  of  Multinational  Programs^' 

Mpjor  Michael  U  Rendine,  USAF 

Coproduction  of  a  mofor  military  system  by  two  orjatofe  governments  is  an 
attractive  method  of  negating  some  of  the  ef^ct'‘^funfavorable  economic  or 
political  conditions.  One  such  prograrn  has' been  the  successful  coproduction  of 
the  F-16.  The  author  discusses  the  cntical  financial  challenges  of  the  F-16  pro¬ 
gram. 


1 27  ^Overview  of  Cost  Analysis  in  the  Navy  /  fohn  S.  Nieroski 

Carl  R.  Wilhoum 

The  authors  describe  the  role,  missioth-ritt^p^nctions  of  the  Resource  Analysis 
Branch  (Op-96D)  as  a  participant  in  the  Navy  weapon  system  acquisition  review 
process  mid  Cost  Analysis  Program. 

</ 

139  The  Multicountry,  Muldperiod^Payments  Problem^  t^.^rfidmond^ 


Multinational  development  and/or  production  of  a  majgr-iiefense  system  car¬ 
ries  with  it  a  number  of  problems  that  do  not  exist  in  single-country 
developments.  The  authors  discuss  the  NATO  ESA  program,  arguing  that  the 
present  structure  for  equitable  payment  for  costs  is  unfair  to  most  of  the  par¬ 
ticipating  governments. 


1 4,  cr 'Electing  Estimating  Techniques  Using  Historical  Simulation  , 

Charles  A.  Graver 


There  are  a  number  of  techniques  that  may  be  used  in  estimating  the  cost  of  a 
planned  system.  The  author  discusses  some  of  the  problems  associated  with  se¬ 
lecting  from  among  these  techniques,  and  presents  historical  simulation  as  a 
means  by  which  these  problems  may  be  addressed  by  the  analyst. 
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from  the  editor  .  .  . 

Ask  anyone  involved  in  the  development  and  acquisition  of  military  systems 
to  list  the  major  issues  in  that  business,  and  you  will  find  cost  at  or  near  the  top. 
Hardly  any  decision  is  made  in  a  development  program  without  some  considera¬ 
tion  being  given  to  cost,  whether  it  be  cost  to  develop,  cost  to  buy,  cost  to  sup¬ 
port,  or  cost  to  maintain.  Unless  those  cost  questions  are  answered  adequately, 
the  program  is  in  trouble. 

The  significance  of  cost  in  the  acquisition  process  prompted  many  of  our 
readers  to  call  or  write  suggesting  that  we  publish  more  material  on  cost  analysis 
and  cost  estimating.  That  was  the  origin  of  the  idea  for  a  special  cost  analysis 
issue  of  Concepts,  the  fruit  of  which  you  now  hold  in  your  hands.  Our  objective 
is  twofold.  First,  we  want  to  provide  useful  information  to  those  people  actually 
charged  with  cost  analysis  responsibilities.  Second,  we  want  to  provide  insight  in¬ 
to  the  workings  of  the  cost  analysis  discipline  for  those  who,  though  not  cost 
analysts  themselves,  must  depend  to  a  great  extent  on  the  results  of  the  cost 
analyst's  work. 

We  of  the  editorial  staff  of  Concepts  want  to  express  our  appreciation  for  the 
invaluable  assistance  provided  us  by  the  OSD  Cost  Analysis  Improvement 
Group,  without  which  this  project  would  never  have  come  to  fruition.  We  are 
particularly  indebted  to  Mrs.  Geri  Asher,  who  worked  with  us  from  the  very 
beginning  of  the  project  to  help  generate  material,  and  then  to  provide  what 
turned  out  to  be  a  great  deal  of  follow-up  work.  The  guidance  and  support  given 
us  by  the  CAIG  and  Mrs.  Asher  serve  to  point  up  the  seriousness  with  which 
DOD  approaches  the  issues  of  cost  analysis  and  cost  estimating. 

Of  interest  to  the  cost  analysis  community  is  the  recent  news  that  an  Institute 
of  Cost  Analysis  is  being  formed  to  develop  and  implement  a  Cost  Analyst  Pro¬ 
gram  for  both  government  and  private-sector  cost  analysts.  The  current  concept 
is  to  have  cost  analyst  standards  published  and  achievement  programs  estab¬ 
lished.  As  with  news  of  the  establishment  of  any  new  professional  development 
association  related  to  acquisition  management,  we  will  provide  more  details  in  a 
future  issue  of  DSMC's  bimonthly  newsletter  Program  Manager. 

It  is  with  a  mixture  of  sadness  and  elation  that  we  find  ourselves  forced  to  say 
goodbye  to  one  of  the  mainstays  of  the  publications  staff.  Susan  Pollock  has  been 
the  Editorial  Assistant  here  since  1977  and  has  not  only  seen  the  Publications 
Directorate  grow,  she  has  been  in  large  measure  responsible  for  that  growth.  She 
provided  much  of  the  initiative,  drive,  and  talent  that  it  has  taken  to  improve  the 
quality  of  DSMC  publications.  Without  her  competent  assistance,  it  would  have 
been  impossible  for  us  to  have  come  so  far  in  such  a  short  time.  Unless  we  can 
quickly  convince  her  that  there's  more  to  life  than  a  higher  salary  and  better  op¬ 
portunities  for  advancement  (fat  chance),  this  will  be  Susan's  last  issue  of 
Concepts.  To  her,  we  wish  the  best  for  the  future.  As  for  us  ...  it  looks  like  a 
long,  hot  summer. 


Improving  Cost 
y  Estimating  in  the 

*  Department  of  Defense 

Milton  A.  Margolis 

The  accuracy  and  realism  of  weapon  systems  cost  estimates  are  prerequi 
sites  for  the  efficient  functioning  of  the  Defense  Systems  Acquisition  Review 
Council  (DSARC),  an  advisory  body  to  the  Secretary  of  Defense  on  all  major 
defense  systems  acquisition  programs  and  related  policy.  At  each  major  program 
decision  point,  all  aspects  of  a  program  are  reviewed  by  the  DSARC.  Programs 
breaching  established  cost  thresholds  are  cause  for  interim  DSARCs.  According¬ 
ly,  emphasis  on  improving  service  cost  estimates  continues  to  receive  high  DOD 
priority.  Taking  the  lead  in  this  effort,  the  OSD  Cost  Analysis  Improvement 
Group  (CAIG)  exercises  the  dual  functions  of  vigorously  reviewing  service  cost 
estimates  for  the  DSARC,  and  fostering  defense-wide  improvements  in  military 
cost  analysis  capabilities. 

Cost  Analysis  Improvement  Group 

The  CAIG  is  made  up  of  representatives  from  the  Under  Secretary  of  Defense 
(Research  and  Engineering),  and  the  Assistant  Secretaries  of  Defense  (Comp¬ 
troller),  (Manpower,  Reserve  Affairs,  and  Logistics),  and  (Program  Analysis  and 
Evaluation).  Also  represented  are  the  military  departments'  cost  analysis 
organizations  and  the  Joint  Chiefs  of  Staff  (JCS). 

The  military  departments  are  required  to  submit  two  program  cost  estimates 
at  each  DSARC  decision  point.  One  estimate  is  prepared  by  the  relevant  program 
office  and  the  second  by  a  separate  organization  within  the  military  department, 
independent  of  the  program  office. 

Fifteen  working  days  prior  to  the  convening  of  the  DSARC,  the  military 
department  presents  these  estimates  to  the  CAIG.  On  the  basis  of  this  presenta¬ 
tion  and  subsequent  analysis,  the  CAIG  prepares  a  memorandum  for  the  DSARC 
with  the  CAIG  evaluation  of  the  estimates  presented.  Meeting  with  the  DSARC 
principals,  the  CAIG  Chairman  summarizes  the  CAIG  evaluation  and  responds 
to  questions  of  the  DSARC  members.  In  supporting  the  CAIG  review  of  the 
military  department  cost  estimates,  the  CAIG  members  contribute  in  accordance 
with  their  particular  expertise.  The  Office  of  the  Secretary  of  Defense  (Program 
Analysis  and  Evaluation)  analysts  generally  coordinate  overall  analysis  and  the 
preparation  of  the  independent  CAIG  estimate.  The  CAIG  and  military  depart¬ 
ment  independent  assessments  serve  as  a  inducement  to  the  program  manager  to 
estimate  his  program  more  accurately.  They  have  even  served  from  time  to  time 
as  a  basis  for  the  program  manager  to  revise  his  estimate.  However,  the  program 
manager  is  held  responsible  for  his  program,  and  it  is  his  estimates  that  are 
reflected  in  the  DOD  budget  sent  to  the  Congress. 


Milton  A.  Margolis  is  Deputy  Assistant  Secretary,  Kesource  Analysis,  in  the  Office  of  the  Assist¬ 
ant  Secretary  of  Defense  (Program  Analysis  and  Evaluation}.  He  is  also  Chairman  of  the  OSD  Cost 
Analysis  Improvement  Croup.  He  served  previously  as  the  Director,  Cost  and  Economic  Analysis 
Directorate,  OASD(PASrE).  Mr.  Margolis  holds  a  B.A.  degree  from  Columbia  University. 
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TABLE  I 

Major  Weapon  Systems  Acquisition 
Annual  Rate  of  Cost  Growth 


Selected  Acquisition  Reports  (SARs) 

As  of  Date 

Annual  Growth  Rate 

December  1972* 

6.4% 

lune  1974* 

5.2% 

March  1975* 

4.4% 

March  1975 

3.7% 

March  1976 

3.0% 

December  1977 

3.3% 

December  1978 

3.6% 

March  1979 

3.4% 

September  1980 

3.7% 

December  1980 

3.9% 

*  Based  on  development  estimates  which  included  anticipated  inflation;  other 
rates  are  based  totally  on  base  year  dollars. 


Cost  Growth  in  SARs 

Cost  growth  reported  in  the  selected  acquisition  reports  (SARs)  is  some  in¬ 
dication  of  the  success  of  CAIG  activities.  Our  examination  of  SARs  (see  Table  I) 
indicates  that  after  adjustment  for  quantity  change  and  inflation,  DOD  showed 
substantial  improvement  in  controlling  cost  growth  during  the  early  '70s  when 
the  overall  annual  cost  growth  rate  dropped  from  over  6  percent  down  to  3.7  per¬ 
cent.  Since  1975,  the  rate  has  been  maintained  between  3.0  percent  and  3.7  per¬ 
cent,  In  this  last  year,  however,  we  have  seen  cost  growth  begin  to  creep  up 
again.  The  December  1980  SARs  show  annual  rate  of  cost  growth  for  DOD  as  a 
whole  of  3.9  percent.  Cost  growth  is  measured  relative  to  the  development 
estimate  made  at  the  time  a  weapon  system  passes  the  DSARC  II  milestone  and 
enters  full-scale  development. 

Our  analysis  of  these  SAR  reports  also  shows  that,  in  reviewing  the  total  cost 
growth  experienced,  unanticipated  inflation  was  the  single  major  factor  con¬ 
tributing  to  cost  growth  in  DOD's  weapon  acquisitions.  During  the  1975-80  time 
period,  unanticipated  inflation  varied  between  70  percent  and  78  percent  of  total 
cost  growth  (see  Table  II).  Of  this  total  unanticipated  inflation,  the  escalation 
associated  with  the  program  as  originally  conceived  remained  rather  constant, 
around  $25  billion,  until  1979;  since  then  it  has  just  about  doubled  with  the  F-18, 
F-16,  Trident,  XM-1,  and  CG-47  accounting  for  $11  billion  of  the  increase.  The 
escalation  associated  with  program  changes  grew  from  $11  billion  in  March  1975 
to  $84.6  billion  in  December  1980.  Much  of  this  increase  was  due  to  the  inflation 
associated  with  the  significant  increases  in  quantities  in  the  F-18,  F-16,  Trident, 
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II 


and  XM-1  programs.  Both  unanticipated  inflation  and  changes  in  program  quan¬ 
tities  are  aspects  of  a  program  over  which  cost  analysts  and  program  managers 
have  very  little,  if  any,  control. 

Table  HI  provides  a  breakout  of  all  factors  of  cost  growth  as  estimated  by  the 
program  offices  for  the  SAR  programs.  Although  none  of  the  other  factors  of  cost 
growth  have  contributed  to  a  substantial  share  of  the  growth,  in  the  aggregate, 
these  factors  have  become  more  significant  over  the  last  few  years.  Note  that  pro¬ 
gram  offices  attribute  no  more  than  6  percent  of  the  total  cost  growth  to 
estimating  errors. 

The  best  and  fairest  way  to  measure  "real"  cost  growth  is  to  adjust  the 
development  estimate  for  both  escalation  and  quantity  changes  in  calculating  the 
cost-growth  ratio.  By  doing  so,  we  have  eliminated  those  factors  beyond  the  con¬ 
trol  of  the  program  manager  and  are  comparing  the  original  and  current  estimates 
of  the  program  for  the  currently  planned  production  quantities.  Table  IV  shows 
this  adjustment  for  the  1975-80  time  period.  The  "real"  cost  growth  appears  to 
have  remained  relatively  stable  at  about  15  percent  from  1975  to  early  1979,  but 
since  then  has  jumped  to  25  percent. 

The  SAR  is  not  a  perfect  reporting  system.  It  is  essentially  unaudited.  Con¬ 
siderable  flexibility  in  classifying  the  sources  of  growth  is  allowed.  We  have  to  ac¬ 
cept  the  fact  that  it  is  a  temptation  for  program  managers  to  assign  changes  in 
their  costs  to  factors  beyond  their  control,  i.e.  inflation.  It  is  a  convenient  way  to 
avoid  acknowledging  real  cost  growth.  Inflation  can,  in  a  real  sense,  undermine 
the  DSARC's  attempts  to  set  and  hold  to  fiscal  constraints  on  individual  acquisi¬ 
tion  programs. 

In  short,  SAR  data  is  only  a  general  indicator  of  weapon  system  cost  status. 
Although  overshadowed  by  the  impact  of  inflation,  improvements  appeared  to 
have  been  made  in  the  early  '70s,  and  was  stabilized  during  the  mid-'70s;  it  now 
appears  to  be  starting  upward  again.  Clearly,  we  need  to  continue  to  drive  for 
better  cost  data,  more  realistic  cost  estimates,  and  better  cost  management.  The 
CAlG's  role  in  this  effort  is  described  below. 

DOD  Directive  5000.4 

The  existence  of  the  CAIG  was  formalized  in  June  197.3  with  the  issuatice  of 
DOD  Directive  5000.4,  the  organization's  charter.  It  was  reissued  on  October  30, 
1980,  with  changes  consistent  with  the  revised  March  19,  1980,  DOD  Directive 
5000.1,  "Major  Systems  Acquisitions,”  and  DOD  Instruction  5000.2,"  Major 
System  Acquisitions  Procedures."  The  "Criteria  and  Procedures  for  the  Prepara¬ 
tion  and  Presentation  of  Cost  Analyses  to  the  OSD  CAIG"  was  added  as  an 
enclosure.  The  DOD  Directive  5000.4  outlines  the  membership  of  the  CAIG  and 
delineates  specific  responsibilities  that  can  be  summarized  as  follows;  performing 
independent  cost  reviews  in  support  of  DSARCs;  providing  criteria  and  pro¬ 
cedures  for  these  reviews;  establishing  DOD's  cost  analysis  policies  and 
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guidelines;  and  integrating  DOD's  cost  research  program  and  cost  data  collec- 
tion>  storage,  and  exchange. 

The  "independent"  cost  estimate  has  been  given  attention  throughout  DOD 
since  the  establishment  of  the  CAIG.  It  has  been  used  to  describe  a  cost  analysis 
prepared  by  a  separate  organization  and  employing  techniques  different  from 
those  used  by  the  program  manager  in  making  his  estimate  of  weapon  system 
cost.  The  CAIG  has  reviewed  many  military  service  estimates  of  varying  quality 
and  independence.  While  it  is  not  possible  to  claim  a  perfect  correlation  between 
the  two,  in  our  experience  the  less  independent  the  service  estimate,  the  less  it 
represents  a  valid  check  on  the  program  manager's  numbers. 

For  example,  in  those  cases  where  both  the  program  office  and  service  in¬ 
dependent  estimates  were  prepared  by  the  same  people,  there  is  a  strong 
adherence  to  the  same  basic  assumptions  and  a  seeming  effort  to  corroborate 
rather  than  to  actually  test  the  base  estimate.  Experience  has  shown  that  in¬ 
dependence  is  an  essential  requirement  if  future  cost  reviews  are  to  be  of  value  to 
the  DSARC. 

The  CAIG  has  participated  in  over  150  DSARC  reviews  since  1972.  In  88  of 
these  reviews  the  CAIG  has  prepared  its  own  independent  cost  estimates.  These 
have  not  necessarily  been  totally  derived  cost  estimates;  in  many  cases  they 
reflect  adjustments  for  disagreements  with  the  service  estimate  over  the  cost  of 
specific  components  or  subsystems,  assumptions  made,  analytical  techniques 
used,  or  assessment  of  cost  risk.  In  about  25  percent  of  these  reviews  the  CAIG 
has  agreed  with  the  service  independent  estimate,  which  ranged  from  20  percent 
below  to  35  percent  above  the  program  office  estimates.  Figure  1  presents  a  com¬ 
parison  of  the  ratio  between  the  CAIG  and  program  office  estimates,  and  the 
CAIG  and  the  independent  cost  estimates. 

These  ratios  over  the  8  years  the  CAIG  has  been  in  existence  have  remained 
fairly  constant.  Both  distributions  are  skewed  to  the  right;  that  is,  the  CAlG's 
estimates  have  tended  to  be  higher  than  either  the  program  office  or  the  service 
independent  estimates,  and  on  the  whole  agree  more  closely  with  the  latter.  The 
CAIG  and  service  independent  estimates  are  generally  based  on  techniques  that 
rely  heavily  on  actual  costs  of  similar  prior  programs  that  have  experienced  the 
normal  delays,  problems,  mistakes,  and  redirection  in  their  development.  The 
program  manager  projections  are  generally  a  product  of  industrial  engineering 
judgments  made  by  the  contractors  involved  with  the  program,  who  are  natural¬ 
ly  in  an  advocacy  role. 

In  a  number  of  cases  where  CAIG  has  differed  significantly  from  the  program 
manager,  the  DSARC  has  requested  that  further,  more  comprehensive  analyses 
be  performed.  For  example,  in  the  case  of  the  Harpoon  anti-ship  cruise  niissile, 
the  CAIG  report  of  June  24,  1974,  projected  the  total  planned  production  missiles 
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would  run  20  percent  higher  than  the  program  manager's  estimate.  It  also  iden¬ 
tified  a  shortfall  in  the  FY7S  funds  budgeted  for  the  Harpoon  pilot  production. 
The  DSARC  Chairman  requested  the  program  manager,  working  with  the  Navy 
independent  estimating  group  and  the  CAIG,  "to  more  fully  assess  projected  cost 
growth  and  its  impact  on  later  year  procurement."  That  study,  completed  in 
November  1974,  indicated  that  procurement  costs  were  underestimated  by  more 
than  50  percent,  and  resulted  in  the  program  manager  revising  his  program  costs 
upward  by  about  40  percent.  To  stay  within  his  FY75  budget,  the  program 
manager  also  had  to  shift  the  completion  of  50  of  the  150  missiles  in  FY75  into  the 
FY76  budget.  The  CAIG  forced  the  cost  issue  into  the  open  and  there  was  an  ad- 
mision  by  all  parties  that  the  program  was  underfunded. 

Another  example  of  the  CAIG  analysis  that  significantly  affected  OSD 
decision-making,  was  a  cost  comparison  of  the  Pratt  and  Whitney  FlOO  and  the 
General  Electric  FlOl-DFE  aircraft  engines.  For  several  years,  the  FlOO  engine  in¬ 
stalled  on  the  F-15  and  F-16  fighter  aircraft  experienced  serious  reliability  prob¬ 
lems.  Although  significant  progress  was  made,  at  great  expense,  to  correct  these 
problems,  the  Air  Force  in  1979  asked  General  Electric  to  develop  a  possible 
substitute  engine.  GE  proposed  the  FlOl-DFE,  a  derivative  of  an  engine  originally 
developed  for  the  B-1  bomber.  The  FlOl-DFE,  although  heavier  than  the  FlOO, 
contains  fewer  parts  and  was  recommended  by  GE  as  a  more  durable  engine  that 
would  be  less  expensive  to  operate. 

When  the  Air  Force  presented  a  proposal  to  substitute  the  FlOl-DFE  for  the 
FlOO  on  the  last  638  F-16s,  the  CAIG  was  asked  to  perform  a  comparative  life- 
cycle-cost  analysis  of  the  two  engines  for  the  F-16.  The  CAIG  found  that  although 
operating  and  support  costs  would  be  significantly  lower  on  the  FlOl-DFE,  these 
savings,  on  a  life-cycle  basis,  could  not  offset  the  much  higher  development  and 
procurement  costs  associated  with  that  engine.  In  fact,  in  FY  1981  dollars  the  ad¬ 
ditional  life-cycle  costs  of  the  FlOl-DFE  vs.  the  FlOO  engine  would  be  over  one- 
half  billion  dollars.  When  costs  were  discounted  at  the  10  percent  rate,  the  addi¬ 
tional  costs  were  almost  three-quarters  of  a  billion  dollars. 

As  a  result  of  this  study,  OSD  decided  not  to  approve  the  substitution  of  the 
FlOl-DFE  for  the  FlOO  on  the  F-16,  but  resources  were  included  in  the  budget  to 
continue  the  engine  development  for  possible  use  on  future  weapon  systems. 

Another  recent  CAIG  analysis  that  contributed  to  OSD  decision-making  was 
the  review  of  the  Navy's  F-18  program.  As  part  of  the  DSARC  III  Review,  the  ac¬ 
quisition  executive  established  a  DOD  Review  Group  on  June  30,  1980,  to 
evaluate  the  technical,  operational,  and  cost  status  of  the  F-18  program.  This 
review  was  designed  to  provide  the  Secretary  of  Defense  with  the  information 
necessary  to  make  an  F-18  production  decision,  and  to  permit  him  to  respond  to 
questions  from  the  President  and  the  Congress  on  the  program. 
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Cost  reports  indicated  that  the  Navy's  current  cost  estimates  for  the  F-18  pro¬ 
gram  had  almost  doubled  since  the  November  1975  DSARC  II.  The  CAIG's 
evaluation  presented  to  the  DSARC  in  the  late  fall  of  1980  showed  that  over  two- 
thirds  of  the  increase  was  caused  by  unexpected  inflation,  and  the  remaining  one- 
third  was  due  to  configuration  uncertainty,  schedule  changes,  and  underestima¬ 
tion  of  equipment  costs.  This  CAIC  cost  analysis  played  an  important  role  in  the 
decision-making  process  on  the  future  of  the  F-18  production  program. 

As  mentioned  above,  an  enclosure  has  been  added  to  the  reissued  DOD  Direc¬ 
tive  5000.4,  "OSD  Cost  Analysis  Improvement  Group,"  containing  the  "Criteria 
and  Procedures  for  the  Preparation  and  Presentation  of  Cost  Analyses  to  the 
OSD  CAIG."  It  addresses  the  objective  of  the  CAIG  presentations  and  organiza¬ 
tional  responsibilities.  It  describes  the  scope  of  the  independent  analysis,  the 
preferred  analytical  methods  to  be  used,  and  the  content  of  the  presentation  of 
the  cost  results  to  the  C  \IG.  It  also  establishes  the  procedures  to  be  followed  in 
preparing  for  a  CAIG  presentation.  These  criteria  and  procedures  have  evolved 
over  the  8  years  the  CAIG  has  been  in  existence  and  represent  the  recommenda¬ 
tions  of  the  DOD  cost-analysis  community. 

The  preferred  analytical  techniques  to  be  used  in  making  independent  cost 
estimates  of  R&D,  investment,  and  operating  and  support  costs  depend  on  the 
stage  of  the  acquisition  cycle  that  the  weapon  system  is  in  when  the  estimate  is 
made.  Until  sufficient  actual  costs  of  R&D  hardware  are  available,  the  CAIG  en¬ 
courages  the  use  of  the  parametric  costing  techniques  for  the  R&D  and  invest¬ 
ment  estimates.  Cost  estimates  prepared  for  DSARCs  I  and  11  should  be  based  on 
parametric,  analog,  and  engineering  techniques  depending  on  the  extent  of  data 
available.  At  DSARC  III,  projection  from  cost  actuals  should  be  the  primary 
basis  for  the  cost  estimate.  In  all  cases,  the  preferred  approach  is  to  use  several 
cost-estimating  methods  to  support  the  independent  acquisition  cost  estimate. 

For  operating  and  support  costs,  much  of  our  current  data  is  still  based  on 
planning  information  rather  than  actual  consumption.  This  presents  many  prob¬ 
lems  in  developing  detailed  operating  and  support  cost  estimates.  As  VAMOSC 
becomes  a  viable  reporting  system,  both  operating  and  support  costs  and  other 
associated  data  will  become  available,  allowing  the  services  to  make  better 
judgments  of  the  correlation  between  operational  concepts,  design 
characteristics,  reliability,  maintainability,  and  so  on,  and  their  impact  on  costs. 
The  CAIG-published  Aircraft  Operating  and  Support,  Cost  Development  Guide, 
April  15,  1980,  updates  and  expands  the  May  1974  guide.  It  is  applicable  to  any 
operating  and  support  cost  analysis  performed  during  the  acquisition  process,  in¬ 
cluding  cost  effectiveness  and  trade-off  studies.  The  guidance  is  directed  at  costs 
used  in  the  acquisition  program  decisions  and  is  not  the  same  as  one  necessarily 
would  use  for  total  program  costs  or  budget  estimates.  The  guidance  provided  in 
this  document  will  be  expanded  to  other  weapon  classes,  and  is  expected  to  be 
issued  in  the  near  future. 
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Progress  to  Date 

The  CAIG  has  been  in  existence  for  a  little  more  than  8  years  and  in  that  time 
there  have  been  substantial  improvements  made  in  the  quality  and  credibility  of 
cost  estimates  supplied  to  Congress.  These  improvements  have  been  achieved 
through  a  variety  of  activities,  among  which  the  most  important  are: 
—Requiring  independent  cost  estimates  from  the  military  departments; 

— Sponsoring  contractor  and  in-house  research  studies  aimed  at  improving  cost 
estimating  techniques; 

— Collecting  and  processing  a  consistent,  comparable  data  base  for  cost  analysis 
purposes; 

—Widening  the  dissemination  of  actual  data  usable  for  cost  analysis; 

— Developing  the  expertise  and  data  base  to  deal  with  analysis  of  multinational 
acquisitions. 

Although  defense  acquisition  programs  still  exhibit  constant-dollar  cost  growth, 
the  ongoing  CAIG  effort  has  contributed  substantially  to  reducing  the  average 
annual  rate  of  cost  growth  prevalent  in  1972.  Continued  emphasis  on  the  present 
cost  improvement  program,  along  with  new  initiatives  in  the  operating  and  sup¬ 
port  cost  area  to  improve  and  give  more  visibility  to  life-cycle-cost  estimates,  is 
providing  the  opportunity  for  even  more  significant  progress  in  defense  weapon 
sytems  cost  estimating.  11 
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In  the  demanding  business  of  defense  systems  acquisition,  there  is  a  critical 
and  continuing  need  for  the  work  of  capable,  creative  professional  cost  analysts. 
The  Department  of  Defense  cost  analyst  is  an  essential  player  in  support  of  such 
efforts  as  Defense  Systems  Acquisition  Review  Council  (DSARC)  reviews,  force 
planning  studies,  and  service  budget  preparation  activities.  Any  neglect  of  the 
professional  development  of  the  defense  cost  analyst  will  have  severe  conse¬ 
quences  for  these  and  other  functions  that  rely  heavily  on  valid  and  readily 
available  cost  estimates. 

For  the  past  15  years,  the  Department  of  Defense  has  sponsored  annual  cost 
analysis  symposia  to  enhance  the  professional  development  of  cost  analysts  and 
to  promote  more  effective  performance  of  the  cost  analysis  function  within  DOD. 
These  symposia  are  designed  primarily  for  managers  and  practitioners  of  cost 
analysis  throughout  DOD  and  for  members  of  the  research  organizations  that 
support  them.  The  symposia  are  devoted  to  such  topics  as  the  output  of  DOD 
and  contract  cost  research  studies;  defense  or  national  issues  that  have  or  could 
have  a  significant  impact  on  defense  cost  analysis;  and  proposed  or  ongoing  cost 
research  programs. 

The  Fifteenth  Annual  Cost  Analysis  Symposium  was  held  in  October  1980. 
To  help  members  of  the  cost  analysis  community  prepare  for  future  challenges,  a 
general  session  meeting  at  that  symposium  was  devoted  to  a  panel  discussion 
addressing  "Estimating  in  the  '80s. "  I  was  privileged  to  be  a  member  of  that  pam  1, 
which  had  as  its  moderator  Mr.  Leroy  T.  Baseman,  Chairman  of  the  Air  Force 
Cost  Analysis  Improvement  Group.  Joining  me  on  that  panel  were  Mr.  Wayne 
Allen,  Director  of  Cost  Analysis,  Office  of  the  Comptroller  of  the  Army; 
Mr.  Riner  Payne,  then  Deputy  for  Financial  Systems  and  Analysis,  Headquarters 
U.S.  Air  Force;  Mr.  Joseph  T.  Kammerer,  Director  of  Cost  Estimating,  Head¬ 
quarters  Naval  Material  Command;  and  Dr.  Michael  Sovereign,  Director  of 
Special  Projects,  Office  of  the  Assistant  Secretary  of  Defense  (Comptroller). 

This  special  section  of  Concepts  is  devoted  to  highlights  of  that  panel  discus¬ 
sion,  including  some  of  the  formal  presentations  and  a  portion  of  the  transcript 
from  the  question-and-answer  period  that  followed.  1  believe  that  these  highlights 
will  be  of  general  interest  to  those  in  the  cost  analysis  community  as  well  as  to 
others  in  the  defense  acquisition  community  who  want  and  need  a  better 
understanding  of  the  problems  and  challenges  that  face  the  DOD  cost  analyst. 
This  increased  understanding  can  only  lead  to  an  increased  appreciation  for  the 
role  of  the  cost  analyst  in  the  efficient  acquisition  of  defense  systems. 
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hen  I  was  first  asked  to  participate  in  this  panel,  I  couldn't  help  but 
think  back  15  years  to  the  first  cost  symposium  which  we  had  in  the  Institute  for 
Defense  Analysis  (IDA)  building.  At  that  time  the  Air  Force  had  just  launched  a 
formal  program  to  assemble  a  data  base  for  cost  analysis.  We  had,  about  the 
same  time,  launched  an  Air  Force-wide  program  to  develop  cost  estimating  rela¬ 
tionships.  Now,  some  of  you  who  are  about  half  my  age  may  marvel  at  the  lack 
of  sophistication  that  this  represents;  however,  to  us  it  represented  a  giant  step 
forward.  I  think  we've  seen  many  more  examples  of  real  progress  in  the  past  few 
years,  which  brings  me  to  the  first  challenge  that  I  think  we  have  for  the  '80s. 
That  challenge  is  not  to  become  complacent  because  of  the  progress  you  have 
made.  Think  to  the  challenges  ahead.  We  have  to  make  sure  that  our  costing 
capabilities  keep  pace  with  unfolding  developments.  For  example,  as  we  move 
into  the  job  of  costing  satellites  and  space  programs,  we  are  dealing  with  a  dif¬ 
ferent  kind  of  technology  than  we  were  dealing  with  when  we  attempted  to  cost 
airplane  programs.  We  are  dealing  with  something  where  development  never 
ends  in  the  sense  that  each  model  differs  from  its  predecessor. 

We  also  have  the  problem  of  dealing  with  the  interdependency  of  systems  and 
subsystems,  as  illustrated  by  LANTIRN.  This  is  under  development  in  the  Air 
Force  to  be  common  to  the  A-10  and  the  F-16.  Further,  if  it  works  out  well,  we 
hope  to  see  it  extended  into  the  aircraft  of  the  other  services.  Now,  to  the  extent 
that  this  gives  us  open-ended  possibilities,  it  represents  a  challenge  to  the  coster  to 
decide  what  sort  of  assumptions  he  must  make  in  estimating  the  cost  of  producing 
that  item. 

We're  also  in  an  environment  of  many  players  from  outside  DOD.  Several  of 
the  other  speakers  have  addressed  the  role  of  Congress  and  that  of  the  GAO,  as 
they  become  increasingly  involved  in  management  decisions  before  the  fact. 

We  have  certain  unique  conditions  imposed  by  the  emphasis  that  has  been 
placed  in  the  last  5  or  6  years  on  rationalization,  standardization,  and  in¬ 
teroperability.  We  now  have  not  only  our  own  nation  but  other  nations  involved 
in  the  development  and  acquisition  of  programs.  This  condition  presents  a  unique 
challenge  to  the  coster. 

Now,  obviously,  while  we  are  talking  about  the  new  things  that  have  to  be 
addressed,  we  are  also  confronted  with  the  job  of  updating  our  existing  data  bases 
and  updating  our  methodologies  and  techniques.  For  example,  we  have  on  the 
horizon  something  called  "tailored  acquisition."  Some  of  us  have  worked  pretty 
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hard  to  figure  out  exactly  what  that  means,  and  today  it  has  kind  of  an  "Alice  in 
Wonderland"  quality  in  the  sense  that  you  might  go  around  and  poll  this  group 
and  probably  get  50  or  75  different  definitions  of  what  it  means.  But  I  will  assure 
you  that  should  all  or  many  of  the  components  of  what  we  call  "tailored  acquisi¬ 
tion"  mature,  it  will  have  a  rather  profound  effect  on  your  existing  data  bases  and 
on  estimating  the  cost  of  future  systems. 

I  wonder  if  we  are  planning  to  make  optimum  use  of  our  costing  capabilities. 
Almost  every  speaker  has  addressed  the  requirement  to  improve  the  capabilities 
of  our  people  and  to  get  more  costers.  1  would  also  like  to  recognize  the  fact  that 
technical  developments  in  automatic  data  processing  equipment  (ADPE)  are  just 
beginning  to  be  exploited  in  the  sense  that  that  they  offer  us  real-time  interface 
between  the  costers  and  the  managers.  I  think  we  should  address  many  of  our  ef¬ 
forts  to  research  which  would  permit  us  to  further  exploit  ADPE  in  the  cost 
analysis  process. 

I  am  a  strong  supporter  of  recognition  of  cost  analysis  as  a  professional.  There 
are  a  number  of  efforts  underway  to  get  us  our  own  job  series.  I  don't  know  how 
long  that  will  take,  but  I  think  that  it  will  come  eventually.  The  challenge  that  I 
would  like  to  throw  out  in  connection  with  the  job  series  is  to  safeguard  against 
making  cost  analysis  a  cult.  If  this  sounds  to  you  like  I  am  equivocating,  that's  ex¬ 
actly  right,  because  I  think,  in  essence,  what  I'm  trying  to  say  is  that  the  biggest 
challenge  we  had  in  the  '50s,  in  the  '60s,  in  the  '70s,  and  will  have  in  the  '80s,  is  to 
maintain  perspective  in  everything  we're  doing. 

I  challenge  you  to  better  support  the  managers  of  the  organizations  which  you 
serve.  You  must  at  the  same  time  avoid  the  pitfalls  of  playing  politician.  Our  con¬ 
tribution  to  management,  as  I  see  it,  is  to  give  the  decision-maker  an  objective 
readout  of  what  we  think  the  situation  is.  Don't  compromise  yourself  by  rework¬ 
ing  the  arithmetic  in  a  way  that  supports  a  preconceived  solution.  It  is  the 
manager's  job  to  make  the  decision. 

I  suggest  that  what  we  need  to  do  first  as  we  move  into  the  '80s  is  to  devote 
more  effort  to  clarifying  our  objectives.  What  exactly  are  we  trying  to  do  in  cost 
analysis?  Many  of  you  start  from  the  premise  that  it  is  the  coster's  job  to  make  the 
estimate  large  enough  that  the  final  answer  will  not  exceed  it.  But,  we  have  a 
responsibility  to  keep  the  system  honest,  not  just  to  produce  a  number  that  will 
cover  us.  I  think  our  responsibility  is  to  decide  whether  or  not  it  is  appropriate  to 
base  an  estimate  on  history  without  further  recognition  of  how  new  manufactur¬ 
ing  processes,  new  acquisition  concepts,  new  technologies,  etc.,  might  tend  to 
modify  and,  in  some  cases,  reduce  the  historical  costs  that  you  feed  into  your 
estimate. 

My  final  point  is  that  we  should  not  become  complacent  because  we  are  able 
to  acquire  a  cost  analysis  job  series.  That's  good,  and  I  hop>e  it  comes  about 
quickly.  But  I  urge  you  to  address,  in  a  realistic  fashion,  the  kinds  of  interfaces  we  ~ 
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need  to  accomplish  in  the  cost  analysis  community  with  the  budgeteers,  with 
managers,  with  the  GAO,  and  with  the  OSD  Cost  Analysis  Improvement  Group 
(CAIG). 

In  summary,  I  would  like  to  remind  you  that  our  biggest  challenge  is  to  keep 
up  the  good  work.  Don't  lose  the  momentum;  pick  it  up.  Don't  rest  on  your 
laurels  because  we  have  developed  estimating  techniques  and  data  bases  that 
have  served  us  well.  Maintain  perspective  in  cost  research  and  in  the  concepts  and 
objectives  that  you  apply  to  your  cost  analysis  tasks.  Improve  your  coordination 
with  other  organizational  elements  that  may  have  different  functional,  but 
related  responsibilities.  || 
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My  discussion  of  the  challenges  of  the  '80s  will  concentrate  on  five  areas 
in  which  we  need  to  improve  in  cost  analysis.  This  perspective  has  evolved  from 
experience  in  my  present  position,  which  serves  as  the  focal  point  for  cost 
analysis  in  the  Naval  Material  Command.  In  the  day-to-day  business  with  our 
subordinate  commands,  the  Naval  Air  Systems  Command,  the  Naval  Sea 
Systems  Command,  and  the  Naval  Electronic  Systems  Command,  which  are  the 
Naval  systems  commands  involved  in  major  weapon  systems  acquisition,  there 
are  several  problem  areas  in  cost  estimating  that  continually  arise.  There  are  in¬ 
flation,  subcontractor  costs,  cost  estimate  tracking,  cost  management  and  cost 
control,  and  the  career  development  of  cost  analysts.  Let's  look  more  closely  at 
each. 

Inflation 

We  know  that  inflation  accounts  for  the  largest  portion  of  cost  growth  during 
weapon  system  development.  The  primary  reason  for  this  is  that  we  continually 
underestimate  inflation  rates  used  for  projecting  future-year  prices.  One  who  is 
not  familiar  with  the  implications  of  this  procedure  might  think  that  it  really 
doesn't  matter  what  inflation  rates  are  used  to  project  prices  5  or  6  years  hence 
because  DOD  funds  are  appropriated  yearly.  The  truth  is  that  out-year  projec¬ 
tions  of  inflation  rates  do  have  a  significant  impact  on  the  budget  year  funds  re¬ 
quested.  The  budget  for  fiscal  year  1981  includes  what  is  supposed  to  be  DOD's 
best  estimate  to  procure  ships,  aircraft,  and  missiles  appropriated  by  the  Con¬ 
gress  for  that  fiscal  year.  This  means  that  sufficient  funds  must  be  requested  to 
cover  expenditures  over  the  next  8  to  10  years  in  the  case  of  ships,  4  to  6  years  for 
aircraft,  and  somewhat  less  for  missiles.  If  inflation  exceeds  that  projected  in  the 
funds  appropriated,  DOD  cannot  plan  on  receiving  additional  funds  to  cover  the 
inflation.  In  the  case  of  ships,  since  most  contracts  are  structured  with  a  contract 
escalation  clause  keyed  to  a  Bureau  of  Labor  Statistics  (BLS)  index,  there  is  a  line 
item  in  the  following  year's  budget  to  recoup  funds  in  excess  of  those  budgeted  for 
contract  escalation.  Such  is  not  the  case  for  aircraft,  missiles,  and  other  equip¬ 
ment.  Consequently,  inflation  rate  projections  are  an  important  part  of  the  proc¬ 
ess  of  developing  cost  estimates.  However,  making  such  projections  is  not  part  of 
a  DOD  cost  estimator's  job.  It  may  sound  silly,  but  such  is  the  case. 

Present  rates  used  in  DOD  are  developed  by  the  Office  of  the  Secretary  of 
Defense  under  the  direction  of  the  Office  of  Management  and  Budget  (OMB). 


Joseph  T.  Kammerer  is  Director,  Cost  Analysis  Division,  Headquarters  Naval  Material  Com- 
mand,  and  Advisor  to  the  Chief  of  Naval  Material.  He  u»as  the  first  CNO/SECNAV  Advisor  for 
Resource  Analysis  from  1971  to  1976.  Mr.  Kammerer  holds  B.S.  and  M.S.  degrees  in  engineering  from 
the  Umversity  of  Maryland,  and  an  M.S.  degree  in  systems  analysis  from  the  University  of  Rochester. 
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OMB's  guidance  is  usually  based  on  the  projected  gross  national  product  (GNP) 
deflator  which  reflects  the  administration's  view  of  what  will  happen  to  the 
economy  if  the  administration's  planned  policies  are  implemented.  This  pro¬ 
cedure  used  over  the  past  few  years  has  resulted  in  low  cost  estimates  in  DOD's 
budget.  According  to  OSD's  assessment  of  the  December  1979  selected  acquisi¬ 
tion  reports  (SARs)  sent  to  the  Congress  in  March  1980,  48  percent  of  the  total 
program  cost  of  the  55  SAR  systems  represents  an  allowance  for  inflation.  Ap¬ 
proximately  72  percent  of  this  is  unanticipated  inflation  that  was  not  recognized 
when  the  development  estimate  was  originally  prepared.  Is  it  possible  to  do  a  bet¬ 
ter  job  of  projecting  inflation  rates  for  defense  expenditures,  and  should  we? 

Many  cost  analysts  shy  away  from  projecting  the  impact  of  the  economy  on 
future  defense  expenditures.  The  excuse  is  often  given  that  such  projections  are 
too  subjective,  too  uncertain,  and  would  be  viewed  as  guesswork.  The  current 
procedure  also  allows  us  to  conveniently  place  the  blame  for  cost  growth  on  infla¬ 
tion,  an  economic  force  beyond  our  control.  However,  there  are  alternative 
views  which  appear  much  more  sensible  for  DOD  and  for  the  profession  of  cost 
estimating  and  cost  analysis. 

The  inflationary  effects  of  defense  programs  are  too  important  to  ignore  or  to 
pass  off  the  responsibility  for  projecting  and  monitoring  these  effects  to  a  second 
or  third  party.  In  addition,  the  actual  price  increases  which  are  the  result  solely  of 
market  and  economic  conditions  will  vary  from  company  to  company  and  even 
product  to  product.  Part  of  the  job  of  the  cost  estimating  and  cost  analysis  com¬ 
munity  should  be  to  closely  monitor  these  trends  on  a  weapon-system  basis.  Only 
then  will  we  have  a  more  realistic  picture  of  what  has  happened  to  prices,  what  is 
happening  and  what  is  likely  to  happen  in  the  future. 

To  assist  in  his  monitoring  and  projecting  of  inflation  on  a  weapon-system 
basis,  the  cost  analyst  should  make  use  of  the  macro-economic  forecasting  serv¬ 
ices  available  from  several  reputable  economic  forecasting  firms.  These  experts, 
who  are  involved  in  projecting  the  overall  economic  conditions  likely  to  prevail 
in  the  future,  have  developed  very  sophisticated  models  that  simulate  economic 
interactions  resulting  from  economic  policies,  legislation,  and  >he  worldwide 
economic  state-of-affairs.  It  is  true  that  results  of  these  macio-economic  models 
are  highly  dependent  on  key  assumptions  concerning  the  lezel  of  federal  spend¬ 
ing,  and  whether  or  not  legislation  such  as  the  Kemp-Roth  tax  laws  are  passed. 
However,  as  in  all  analysis,  a  complete  communication  of  the  underlying 
assumptions  should  be  required.  The  impact  of  any  assumptions  can  be  easily 
tested  in  the  available  macro-economic  models. 

During  the  summer  of  1980,  the  Defense  Science  Board  was  especially  inter¬ 
ested  in  communicating  to  DOD  the  impact  of  inflationary  effects  in  the 
aerospace  industry  during  1980.  The  Board  not  only  made  inquiries  through  their 
industrial  representatives,  but  the  Chairman  also  requested  the  opinion  of  the 


Estimating  in  the  '80s 


25 


Chief  of  Naval  Material.  In  developing  our  response  to  the  Defense  Science 
Board,  we  enlisted  the  macro-economic  services  of  one  of  the  major  economic 
forecasting  firms.  Using  the  results  of  the  macro-economic  models  and  devel¬ 
oping  some  simple  models  that  would  reflect  what  was  happening  in  ships, 
missiles,  aircraft,  and  research  and  development  costs,  we  were  able  to  compare 
what  was  actually  happening  in  1980  to  what  we  had  projected  using  the  latest 
OSD/OMB  indices.  We  extended  the  analysis  to  make  use  of  the  forecasting 
firm's  long-term,  10-year  forecast  to  make  comparison  with  the  OSD/OMB  in¬ 
dices  for  future  years.  We  then  calculated  the  dollar  impacts  of  these  out-year 
projections  on  our  near  term  1980  budget  supplemental  and  1981  budget  amend¬ 
ment  requirements. 

The  results  of  the  exercise  are  shown  in  Figures  1  through  3.  Figure  1  shows 
the  firm's  projections  for  aircraft  (APN),  missiles  (WPN),  and  other  equipment 
(OPN)  compared  to  the  OSD/OMB  procurement  index.  Figure  2  shows  the  ship 
construction  index  (SCN)  compared  to  the  OSD  index  for  shipbuilding.  Figure  3 
shows  the  firm's  projection  for  research  and  development  (RDT&E)  compared  to 
the  OSD/OMB  index  for  research  and  development.  Figures  4-6  show  the  actual 
inflation  rates  shown  in  Figures  1-3. 

We  found  that  the  firm's  projections  closely  approximated  the  OSD  projec¬ 
tions  for  1980  and  1981,  but  beyond  1981,  the  firm's  prc  ctions  were  signifi¬ 
cantly  above  the  OSD/OMB  rates.  This  pattern  of  low  inflation  rates  in  the  out- 
years  has  remained  consistent  over  the  last  decade.  The  low  rates  in  the  out-years 
create  a  very  real  near-term  problem  because  they  cause  shortfalls  in  the  near¬ 
year  appropriations  resulting  from  the  application  of  outlay  rates.  The  funding 
impact  for  the  Navy's  FY-80  Supplemental  request  and  the  FY-81  Budget  Amend¬ 
ment  are  shown  in  Figures  7  and  8.  Using  the  forecasting  firm's  rates  instead  of  the 
OSD/OMB  rates  yields  a  difference  in  the  five  appropriation  accounts  of  $985.5 
million  for  the  FY-80  Supplemental  and  $1,190.5  million  for  the  FY-81  Budget 
Amendment.  This  is  a  significant  amount  of  money  which,  i.*  not  recouped,  will 
leave  no  alternative  but  to  reduce  the  quantities  of  weapon  systems  authorized  by 
the  Congress  for  procurement. 

We  can,  and  we  should,  do  a  better  job  of  projecting  inflation  in  weapon 
system  cost  estimates.  We  should  monitor  inflationary  trends  on  a  weapon 
system  component  level,  e.g.  aircraft  engines,  airframe,  avionics.  The  services 
should  also  encourage  OSD  to  propose  new  ways  of  projecting  inflation  rates  to 
OMB.  The  present  techniques  of  deriving  the  rates  from  the  GNP  deflator  is  ob¬ 
viously  not  giving  us  good  projections.  Regardless  of  whether  or  not  OSD  suc¬ 
ceeds  in  making  such  a  proposal  to  OMB,  the  services  should  direct  their  cost 
estimators  and  defense  contractors  to  track  actual  inflationary  impact  by  weapon 
system  component.  This  >«  so  important  because  double-digit  inflation  will  most 
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FIGURE  4 

APN/WPN/OPN  Inflation  Rates  % 
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RDT&E  InfSdtion  Rates  % 
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FIGURE  8 

Impact  on  Navy's  FY-81  Budget  Amendment  Request 
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likely  persist  throughout  the  1980s.  If  there  is  to  be  "real  growth"  in  defense  ex¬ 
penditures  as  planned,  DOD  cost  estimators  must  be  able  to  tell  top  management 
the  right  amount  of  money  to  budget  for  weapon  system  procurements  to  permit 
the  real  growth  to  occur.  Former  Secretary  of  Defense  James  Schlesinger,  in  a 
Wall  Street  Journal  editorial  of  October  24,  1980,  said  that  it  would  take  $10 
billion  more  to  execute  the  President's  program  in  the  FY-81  budget  submitted  to 
Congress  in  January  1980.  Although  inflation  between  January  and  October  was 
probably  more  than  anticipated  by  most  economists,  the  budget  reflected  overly 
optimistic  projections.  A  better  job  can  be  done  and  DOD  cost  estimators  should 
lead  the  crusade. 

Subcontractor  Costs 

Certainly  one  of  the  challenges  of  the  '80s  is  for  DOD  cost  estimators  to  pay 
more  attention  to  subcontractor  costs.  The  "make  vs.  buy"  decisions  being  made 
by  our  prime  contractors  are  having  a  larger  impact  on  cost  growth  than  ever 
before.  The  complexity  of  today's  weapon  systems  makes  it  desirable  for  a  prime 
contractor  to  seek  the  services  of  the  increasing  number  of  specialty  shops.  A 
technique  that  is  often  employed  by  the  prime  contractor  in  attempting  to  get  the 
best  arrangement  with  subcontractors  is  to  give  a  firm  order  with  options  for 
future  buys.  The  prime  contractor  often  attempts  to  fix  a  price  at  the  time  the  ini¬ 
tial  orders  are  placed.  As  the  procurement  proceeds  and  the  prime  contractor 
does  not  establish  a  second  source,  the  subcontractor  often  takes  the  position  of  a 
sole-source  supplier.  When  the  options  are  completed  from  the  initial  buy,  he 
raises  his  prices  significantly.  Such  was  the  case  in  both  the  F-15  and  F-18  pro- 
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curements  and,  with  the  inflationary  trends  projected  for  the  1980s,  we  can  ex¬ 
pect  such  price  increases  from  subcontractors  and  vendors  to  become  com¬ 
monplace.  Often  when  the  subcontractor  has  agreed  to  a  price,  he  may  find 
himself  making  no  profit  or  accepting  a  loss  in  the  last  year  of  the  option.  He  will 
then  attempt  to  correct  his  position  with  price  increases.  The  result  seen  by  the 
prime  contractor  and  the  DOD  program  manager  is  a  significant  increase  which 
comes  as  a  surprise  to  DOD  managers.  Cost  estimators  must  make  DOD 
managers  aware  of  this  situation  and  budgetary  estimates  should  be  adjusted  to 
reflect  subcontractor  relationships  with  prime  contractors.  A  closer  monitoring 
of  subcontractor  costs  must  remain  a  top  priority  for  DOD  cost  estimators.  In  ad¬ 
dition,  our  cost  collection  systems  should  be  adjusted  so  that  monitoring  of  sub¬ 
contractor  costs  is  enhanced. 

Cost  Estimate  Tracking 

Whenever  a  Navy  weapon  system  program  has  significant  cost-growth  prob¬ 
lems,  the  questions  from  top  management  are,  "What  happened?  Where  did  we 
go  wrong  in  estimating  the  costs?"  The  answers  are  often  not  readily  available 
and  the  reason  is  the  lack  of  good  cost  estimating  documentation.  In  order  to 
track  cost  estimates,  good  documentation  is  essential.  Even  when  there  is  fairly 
good  documentation,  the  question  can  often  not  be  answered  satisfactorily 
because  cost  estimators  do  not  track  cost  performance  very  well.  We  need  cost 
analysts  who  can  understand  the  implications  of  not  meeting  performance 
specifications  and  who  understand  the  cost  uncertainties  associated  with 
technical  performance  characteristics.  We  need  cost  analysts  who  will  follow  the 
details  of  engineering  changes  made  throughout  development  and  early  produc¬ 
tion  so  that  cost  estimates  can  be  tracked  better.  The  tracking  of  cost  estimates 
can  be  a  full-time  job  requiring  intelligent  analysis  in  terms  of  what  is  significant 
and  what  is  not  and  the  impact  on  future  buys.  To  do  this  requires  a  good 
unde,  ...  iding  of  how  the  contractor  incurs  cost  and  how  he  tracks  cost.  In  track¬ 
ing  costs,  we  need  to  be  able  to  isolate  production  rate  effects  to  be  able  to  better 
articulate  the  impact  of  ineffecient  utilization  of  defense  facilities  and  resources. 
Certainly  then,  a  challenge  for  the  '80s  is  to  be  able  to  track  costs  better. 

Cost  Management  and  Cost  Control 

Cost  analysis  should  be  used  in  weapon  system  procurement  from  the  plan¬ 
ning  stage  until  the  weapon  system  procurement  is  complete.  It  should  be  used  as 
a  tool  to  enhance  cost  management  and  cost  control.  Navy  experience  shows  that 
program  managers  tend  to  pay  more  attention  to  meeting  technical  performance 
specifications  and  planned  delivery  dates  in  weapon  system  acquisition  than  they 
do  to  cost  management  and  cost  control.  Regardless  of  the  problems  we  have  in 
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obtaining  funds  for  weapon  system  procurement,  we  tend  not  to  be  serious  fn  im¬ 
plementing  good  cost-management  and  cost-control  techniques,  DOD  cost 
estimators  can  and  should  pay  a  vital  role  in  helping  program  and  financial 
managers  to  bring  about  better  cost  management  and  cost  control.  In  developing 
the  initial  cost  estimates  and  tracking  costs  through  program  development,  the 
cost  analyst  can  provide  insight  to  the  program  manager  from  this  valuable  ex¬ 
perience.  He  can  tell  the  program  manager  where  the  soft  spots  and  the  significant 
"cost  drivers"  in  the  cost  estimate  are  in  order  to  direct  his  attention  for  cost 
management  to  the  right  areas.  The  cost  analyst  can  keep  contractor  surprises  to 
a  minimum  by  close  tracking  and  can  suggest  management  reviews  when  his  cost 
analysis  indicates  the  need.  There  is  no  question  that  cost  estimating  and  analysis 
is  applied  to  a  much  greater  degree  at  the  front  end  of  program  acquisition  than  in 
the  later  stages  once  production  has  started.  But,  it  is  in  this  stage  of  program  ex¬ 
ecution  that  cost  control  techniques  should  be  applied  rigorously. 

The  cost  analyst  can  assist  the  contract  {personnel  in  this  stage  in  following 
overhead  costs.  At  one  time,  DOD  was  very  interested  in  tracking  defense  con¬ 
tractor  overhead  charges  and  initiated  the  PIECOST  system  to  assist  in  this  ef¬ 
fort.  Although  PIECOST  fell  out  of  favor  because  of  the  large  resources  needed  to 
implement  the  system,  we  have  seen  overhead  charges  increasing  steadily  since  its 
demise.  Perhaps  we  need  to  take  another  look  at  PIECOST  or  a  derivative  of 
PIECOST  to  help  us  better  track  and  control  overhead  charges.  The  challenge  for 
the  ’80s  is  to  exercise  better  cost  management  to  bring  about  better  cost  control  in 
our  weapon  system  procurements. 

Career  Development 

There  is  no  doubt  that  the  profession  of  cost  analysis  and  cost  estimati.ig  is 
suffering  because  we  cannot  attract  and  keep  the  caliber  of  people  needed  to  do 
the  job  right.  Our  good  people  are  seeking  employment  outside  of  government 
because  we  have  failed  to  provide  them  with  a  good  career  development  pro¬ 
gram.  We  need  to  propose  a  "GS  series"  for  cost  estimators  and  analysts  to  the 
Office  of  Personnel  Management.  The  Army  has  led  the  way  in  pioneering  such 
an  effort  but  they  have  lacked  the  needed  support  of  the  other  services  and  OSD 
in  bringing  about  its  implementation.  If  costs  are  of  primary  concern  in  DOD, 
then  we  must  provide  the  department  with  a  means  for  obtaining  the  best  people 
with  the  right  educational  background  to  do  the  job.  We  need  to  concentrate  on 
the  "multi-discipline"  approach  to  cost  analysis.  The  profession  needs  engineers, 
statisticians,  economists,  operations  research  analysts,  program  analysts,  finan¬ 
cial  management  analysts,  system  analysts,  mathematicians,  computer  experts, 
and  experts  from  related  fields.  Witiiout  the  influence  of  all  these  areas  of  exper¬ 
tise,  the  profession  cannot  function  efficiently.  The  challenge  for  the  '80s  is  to 
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develop  a  meaningful  career  development  program  for  cost  estimators  and 
analysts. 

The  five  areas  discussed  above  represent  specific  areas  in  which  improvement 
in  cost  analysis  is  needed.  They  represent  my  view  of  the  challenge  for  the  '80s. 
But,  we  should  not  stop  at  these  challenges.  By  concentrating  on  objectives  of  the 
Under  Secretary  of  Defense  for  Research  and  Engineering,  it  is  easy  to  see  addi¬ 
tional  challenges  for  the  cost  analysis  community  in  the  '80s. 

Two  objectives  of  defense  investment  for  the  '80s,  as  stated  by  the  Under 
Secretary  of  Defense,  are  to  modernize  U.S.  deployed  defense  systems,  and  main¬ 
tain  leadership  in  the  technologies  critical  to  defense.  Because  of  the  cost  of 
modern  weapon  system  acquisition,  we  will  see  much  more  emphasis  being 
placed  on  modernization  of  existing  weapon  systems  as  opposed  to  acquisition  of 
new  ones.  As  cost  analysts,  we  must  be  ready  to  meet  this  challenge  in  estimating 
modernization  costs.  Within  the  Navy,  estimating  modernization  cos^s  is  a 
weakness,  especially  in  our  Washington-based  system  commands.  Perhaps  we 
should  develop  that  expertise  at  a  field  activity— perhaps  a  Naval  shipyard  or  air¬ 
craft  rework  facility.  We  also  have  a  weakness  in  estimating  the  costs  of  weapon 
systems  which  utilize  new  technologies,  and  in  estimating  the  cost  of  developing 
new  technology.  This  is  definitely  an  area  in  which  cost  analysts  will  find  a 
challenge  in  the  '80s. 

Figure  8  shows  some  management  initiatives  and  objectives  continuing  in  the 
'80s  and  we  need  to  be  prepared  as  cost  analysts  to  assist.  Do  we  really  know  the 
cost  impact  of  increased  competition  in  procurement?  In  most  cases  competition 
usually  costs  more  at  the  outset  to  reap  downstream  benefits.  But,  as  a  communi¬ 
ty,  we  lack  sufficient  data  to  adequately  assess  the  effects  of  competition.  This  is 
certainly  an  area  in  which  cost  research  could  help  us. 

Another  area  requiring  emphasis  by  cost  analysts  during  the  '80s  is  manufac¬ 
turing  technology.  Again,  DOD  cost  analysts  are  not  experts  in  this  area  and  we 
need  to  think  nbout  how  we  are  going  to  get  the  proper  assistance  in  this  area.  Ex¬ 
tending  the  useful  life  of  existing  systems,  which  applies  directly  to  the  Navy’s 
carrier  service  life  extension  program  (SLEP),  creates  a  challenge  for  the  cost 
analyst.  The  life  of  aircraft  carriers  is  being  extended  from  30  to  45  years  in  this 
program  and  the  cost  for  the  first  one  is  in  the  order  of  $500  million.  Navy  cost 
analysts  have  prepared  studies  to  compare  the  costs  of  doing  the  job  in  public  and 
private  shipyards  in  accordance  with  OMB  Circular  A-76.  There  will  be  more 
such  studies  in  DOD  and  the  cost  analysis  community  will  find  it  a  challenge  for 
the  '80s. 

With  more  attention  being  given  to  cooperation  with  U.S.  allies  to  develop 
and  produce  weapon  systems,  we  must  be  prepared  to  pay  more  attention  to  co^ 
producton  costs  and  offset  agreements  and  their  impact  on  our  weapon  system 
prices.  We  are  faced  with  the  challenge  of  efforts  to  shorten  the  acquisition  proc- 
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ess.  This  usually  will  result  in  concurrency  in  development;  we  as  cost  analysts 
must  be  prepared  to  properly  assess  what  concurrency  means  in  terms  of  cost.  If 
technical  problems  occur  during  concurrent  development  that  result  in  increased 
cost  in  production,  the  cost  of  making  the  technical  fix  after  production  has 
started  must  be  assessed.  This  complicates  the  job  for  the  cost  analyst  and  will 
make  him  concentrate  more  on  cost  risk  and  on  reflecting  that  risk  in  terms  of  a 
cost  range. 

The  '80s  present  some  interesting  challenges  for  cost  analysts,  both  from  an 
introsptective,  self-improvement  aspect,  and  from  new  initiatives  and  objectives 
proposed  for  weapon  system  acquisition  and  modernization.  Certainly  the 
challenges  are  there;  finding  the  right  people  with  the  right  background  to  do  the 
quality  work  that  we  all  expect  may  be  the  ultimate  challenge  for  the  '80s.  || 
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East  year  at  this  meeting,  1  gave  a  paper  entitled  "The  Decade  of  The  '70s, 
Improvements  in  Cost  Estimating."  In  it  1  cited  the  development  of  the  Cost 
Analysis  Improvement  Group  (CAIG),  the  selected  acquisition  reports  (SARs), 
the  cost  performance  reports,  the  VAMOSC  system,  the  Bureau  of  Economic 
Analysis  inflation  measurements  of  weapon  systems  and,  in  general,  a  higher 
standard  of  professionalism  in  the  cost  community  in  the  broad  sense.  1  think 
there  really  were  improvements  in  the  '70s.  It  was  the  spreading  of  professional 
knowledge  and  the  acceptance  of  the  computer  as  a  tool  which  were  the  greatest 
forces,  in  my  opinion,  behind  the  improvement  of  the  1970s. 

Well,  what  challenges  are  to  be  met  in  the  '80s  and  how  will  we  meet  them? 
Certainly  they  have  not  changed  in  their  general  scope  since  the  '70s.  First,  we 
must  become  more  professional,  i.e.,  more  thorough,  more  realistic,  more  able  to 
support  the  decision-makers,  and  more  willing  to  admit  that  this  is  a  difficult 
business  and  that  we  don't  have  all  the  answers.  1  think  it  is  amazing  that,  in¬ 
dependently,  all  of  the  five  of  us  raised  the  professionalism  issue  in  our  talks,  and 
I  think  this  is  the  decade  for  us  to  really  make  some  radical,  formal,  and  substan¬ 
tial  improvements  in  the  professionalism  of  the  group.  Secondly,  we  must  use  the 
computer  to  link  the  detailed  knowledge  of  those  out  in  the  field  with  the  review 
and  planning  process  in  headquarters.  As  Mr.  Payne  said,  there  must  be  a  closer 
communication  between  the  field  and  headquarters.  The  computer  and  telecom¬ 
munications  revolution  is  really  the  answer  to  that,  and  1  think  we  will  take  ad¬ 
vantage  of  it  in  the  next  decade.  Sometimes  we  here  in  Washington  are  guilty  of 
thinking  of  both  the  government's  and  the  contractor's  accountants  and  others  in 
the  field  as  enemies  instead  of  potential  allies  and  possible  monitors  of  the 
resource  activities  that  drive  our  estimates.  We  have  got  to  communicate  better 
with  the  field  people.  That's  enough  philosophy. 

I  would  like  to  discuss  four  areas  for  the  '80s  which  I  divide  into  problems  of  a 
conceptual  nature  and  problems  of  an  organizational  nature.  Conceptual  prob¬ 
lems  require  new  or  more  definitive  theoretical  guidance.  Organizational  prob¬ 
lems  are  largely  ones  of  implementation,  but  may  require  substantial  organiza¬ 
tional  change,  which  is  always  difficult.  In  the  conceptual  area,  I  would  like  first 
to  point  my  finger  at  the  fundamental  area  of  cost  estimating  relationships 
(CERs),  which  have  been  a  major  tool  in  the  costing  revolution. 


Dr.  Michael  Soi’ereigii  is  the  Director.  Special  Projects  Group  OASD  (Comptroller).  He 
preciously  sereed  as  Professor  and  Chairman.  Department  of  Operations  Research.  Naval 
Postgraduate  School  and  Operations  Research  Analyst.  OASD  (Systems  Analysis)  Dr.  Sovereign 
holds  a  B  S.  degree  in  chemical  engineering  from  the  University  of  Illinois,  an  M.S.  degree  in  chemical 
engineering  from  Purdue  University,  and  a  Ph.D.  degree  in  industrial  economics,  also  from  Purdue 
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Some  people  refer  to  this  as  the  small-sample  problem.  This  problem  is 
definitely  getting  worse  as  we  are  able  to  afford  a  smaller  and  smaller  number  of 
new  systems  on  a  periodic  basis.  I  t)elieve  this  is  an  even  more  fundamental  prob¬ 
lem  than  the  legitimate  small-sample  problem  in  a  statistical  sense.  In  producing 
CERs  and  then  treating  them  as  statistical  forecasts,  we  are  violating  the  fun¬ 
damental  principle  of  statistics,  which  is  that  our  data  must  be  samples  from  a 
definable,  single  population.  As  Mr.  Payne  just  said,  the  reason  we  design  new 
systems  is  that  they  are  different,  otherwise  we  would  buy  more  of  the  old  ones. 
We  must  be  aware  of  this  problem  and  not  mislead  the  decision-makers  on  the  ac¬ 
curacy  of  our  forecasts.  Well,  what  can  be  done  about  this  problem?  An  extreme, 
or  puristic,  view  would  be  that  we  can  do  nothing  about  it,  and  we  should  just 
ignore  any  statistical  properties  of  the  cost  estimating  relationship.  However,  I 
assert  that  this  is  more  defeatist  than  we  need  be.  We  can  start  by  interpreting  our 
results  as  a  lower  limit  on  possible  errors.  We  should  never  say  how  good  it's  go¬ 
ing  to  be,  only  that  it  may  be  at  least  this  bad  an  estimate.  We  can't  guarantee 
that  we  will  be  as  good  as  we  say  our  confidence  intervals  are,  because  we 
disobey  the  basic  assumptions  of  statistical  sampling.  In  this  respect,  we  have 
been  particularly  careless,  I  think,  with  the  statistical  difficulties  of  combining 
estimates  of  unit  costs  with  the  learning  curve,  which  we  then  apply  together  to 
get  total  program  cost.  The  uncertainty  within  the  total  program  cost  is  much 
larger  than  we  often  convince  the  decision-makers  there  is,  as  borne  out  by 
retrospective  examination  of  the  cost  estimates. 

Another  Approach 

Another  approach  is  simply  to  be  more  careful  in  selecting  the  data  base  we 
use  in  deriving  a  CER  for  a  particular  new  system.  We  can  try  to  have  systems 
that  are  fairly  similar.  They  may  not  be  exactly  the  same  as  the  system  we  are  try¬ 
ing  to  estimate  the  cost  of,  but  let's  at  least  pick  a  data  base  of  similar  systems  for 
CERs.  The  computer  and  the  spread  of  statistical  capability  in  the  profession  has 
made  it  easy  to  derive  CERs  for  individual  new  programs  rather  than  looking  for 
one  CER  for  airframes  that  we  will  use  regardless  of  what  kind  of  airframe  it  is, 
say  a  CX,  or  a  fighter,  or  so  forth.  I  call  this  general  problem  of  data  selection  for 
a  particular  system  the  aggregation  problem  in  forming  a  CER.  It  really  has  two 
dimensions.  One  is:  What  kinds  of  systems  do  1  include  in  the  data  base  for  my 
CER— bomber,  cargo,  or  fighter?  The  other  question  or  dimension  is  at  what 
level  of  system  do  I  attach  this  problem?  What  do  1  derive  the  CER  for — for  the 
whole  aircraft,  for  the  airframe,  for  the  fuselage,  or  for  wing  structures?  In  other 
words,  at  what  level  of  system  or  subsystems  do  1  attack  the  problem?  Attacking 
the  problem  by  subsystem  is  often  profitable  because  the  various  subsystems 
have  differential  maturity. 
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The  cost  of  an  engine  from  an  existing  aircraft  is  well  known.  There  isn't  much 
uncertainty  in  the  mature  subsystem.  We  can  often  get  a  measure  of  uncertainty 
by  breaking  the  system  down.  You  are  all  familiar  with  those  general  questions. 
Typically,  we  address  those  questions  of  what  kinds  of  systems  to  include  in  the 
data  base  and  whether  to  attach  the  problem  at  the  system  or  subsystem  level  intui¬ 
tively.  We  usually  just  use  judgment  in  determining  the  answers  to  those  questions. 

I  believe  that's  a  reasonable  approach  until  our  friends  from  the  audit  agencies 
ask  us  to  document  why  we  selected  certain  systems  to  put  in  the  data  base.  We 
probably  are  going  to  have  to  do  better  than  just  rely  on  judgment  in  determining 
these  answers.  There  are  some  statistical  techniques  becoming  available  which  we 
should  be  working  on  to  address  questions  of  what  can  be  treated  as  one  popula¬ 
tion  from  which  to  draw  a  CER.  There  is  the  Chow  test  for  deciding  if  two 
samples  can  be  treated  as  coming  from  the  same  populations.  There  is  work  going 
on  by  Professor  Ted  Walhenias  of  Clemson.  He  uses  what  is  known  as  the 
Mahalanobis  distance  for  measuring  quantitatively  how  different  various 
weapon  systems  are  from  each  other.  This  can  then  be  used  in  determining  what 
points  to  put  in  the  data  base.  In  general,  data  analysis  techniques  that  require 
fewer  distributional  assumptions  may  be  useful  in  segregating  some  populations. 
However,  whenever  we  use  data-analysis-type  techniques  such  as  stepwise 
regression,  we  have  to  be  very  careful  about  the  statistical  properties  of  the 
resulting  CERs.  In  summary,  I  believe  there  will  be  a  very  useful  interaction 
between  theoreticians  and  practitioners  in  the  area  of  development  of  better 
methods  for  selecting  the  data  base  for  CERs  in  the  1980s. 

Personnel  Costing 

The  second  area  in  which  I  see  some  conceptual  difficulties  and  challenges  is 
that  of  personnel  costing.  By  this  I  mean,  in  the  simplist  sense,  what's  the  cost  p»er 
man-year?  That  seems  to  be  a  fairly  simple  thing,  but  it  gives  us  some  difficulties. 
I  think  we  still  see  bad  decisions  by  decision-makers  who  don't  appreciate  the  full 
cost  of  manpower,  or  people,  if  you  prefer.  Some  of  these  decision-makers  are 
mislead  by  conventions  we  adopt,  such  as  the  CAIG  guidance  that  says  the 
operating  and  support  cost  estimates  we  make  should  use  the  costs  p>er  man  ob¬ 
tained  by  simply  taking  the  total  wage  costs  in  the  service  and  dividing  by  the 
number  of  enlisted  men  in  the  service.  There  is  no  allowance  for  skill  or  rank  in 
those  kinds  of  numbers.  Putting  together  operating  and  support  costs  based  on 
those  kinds  of  costs  per  man  is  misleading.  The  disappearance  of  the  draft  has 
removed  the  last  excuse  for  us  to  disregard  the  cost  of  our  human  capital.  I 
believe  that  unless  we  adopt  the  human  capital  costing  procedures  for  this  area, 
wp  will  find  ourselves  with  truly  catastrophic  situations  in  the  near  future  as  eligi¬ 
ble  populations  decline  and  skill  levels  must  increase.  Yesterday  it  was  pointed 
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out  by  one  of  the  panelists  that  the  number  of  additional  military  manpower 
assumed  by  one  of  the  other  panelists  was  larger  than  the  population  coming  into 
the  eligible  area  for  our  manpower  in  that  5-year  period.  So  we  have  a  real  prob¬ 
lem  which  must  be  met,  I  believe,  by  better  accumulation  of  the  full  costs  of  pro¬ 
curing,  training,  maintaining,  and  supporting  our  personnel,  including  retirement 
costs.  If  we  use  that  total  cost,  we  will  be  able  to  reject  new,  manpower-intensive 
systems  in  favor  of  those  which  require  fewer  and  less-skilled  manpower.  We  will 
be  able  to  show  that  personnel  retention  measures  are  cost  effective  and  avoid 
many  of  the  problems  of  rapid  turnover  that  we  have  had  in  the  military. 

We  must  give  up  our  convenient  but  conceptually  inadequate  idea  that  cost  is 
reflected  in  what  we  pay  people.  Opportunity  cost  is  the  only  appropriate 
measure,  and  productivity  should  be  the  indicator,  not  just  pay.  The  training  that 
we  give  our  personnel  creates  opportunities  for  employment,  or  alternative 
employment,  which  we  should  recognize  in  our  costing  or  we  are  going  to  see  our 
best  people  continue  to  leave  the  military  services.  A  new  manpower  costing 
system  must  first  of  all  reflect  the  costs  of  training,  if  not  the  value.  It  must  reflect 
relative  talent  levels  of  people.  The  real  cost  of  the  Navy's  nuclear  program  is,  I 
believe,  substantially  understated  by  the  fact  we  send  the  best  people  into  that 
program.  Many  other  areas  simply  can't  get  that  same  quality  of  manpower, 
which  they  also  need.  Finally,  a  new  system  must  treat  manpower  costs  as  a 
stream  including  the  recruitment  and  the  retirement  costs.  There  are  difficulties 
conceptually  with  retirement  costs,  including;  What  is  the  real  growth  of  pay 
over  the  next  decade  when  these  people  will  retire?  What  is  the  opportunity  cost 
of  capital  which  should  be  applied  against  any  offsetting  deductions  of  their 
retirement  pay?  This  new  methodology  for  assessing  personnel  costs  should 
enable  decision-makers  to  keep  track  of  the  stock  of  military  human  capital  and 
what  is  happening  to  it,  and  make  better  decisions  on  both  its  use  and  level. 

Implementation  Challenge 

The  second  area  I  would  like  to  talk  about  is  the  implementation  challenge  we 
will  have  in  the  1980s.  I  have  picked  out  two  here  that  are  relevant  to  the  kinds  of 
things  I'll  be  doing.  First  is  a  coordination  of  life-cycle  weapon  system  costs  with 
long-term  budgeting.  We  must  bring  together  the  life-cycle  costing  people  and  the 
budget-analysis  people.  Conceptually,  it  should  be  possible  to  build  budgets  by 
knowing  force  levels  and  acquisition  plans  for  new  systems,  along  with  the  life- 
cycle  costs  of  both  the  new  and  existing  systems.  Indeed,  the  Army,  under  W'ayne 
Allen's  leadership,  is  attempting  to  do  just  this.  In  the  first  issue  of  the  new 
Resource  Management  Journal  the  Army  is  putting  out  a  very  nice  account  of 
what  Wayne  is  trying  to  do.  I  believe  more  connections  can  be  drawn  between 
existing  operating  and  support  cost  reporting  systems  and  the  budget  and  plan- 


» 


40 


Concepts 


ning  processes.  There  is  a  cost-of-ownership  project  going  on  in  OP-96  in  the 
Navy,  under  Irv  Blickstein;  there  is  Charlie  Groover's  Logistics  Resource  Annex 
coming  out  of  OSD(MRA&L);  there  are  the  VAMOSC  systems.  We  need  to  put 
all  of  those  together  to  allow  us  to  be  able  to  bridge  this  long-standing  separation 
of  the  acquisition  and  the  budget-analysis  cost  communities. 

Finally,  I  would  like  to  speak  a  little  bit  to  the  inflation  problem,  which  Joe 
Kammerer  raised,  particularly  his  treatment  of  the  rates  imposed  on  us  as  OSD 
inflation  rates.  Coming  from  the  Comptroller's  Office  where  we  put  these  out,  let 
me  tell  you  that  those  are  not  the  rates  we  would  like  to  see.  Those  are  the  rates 
imposed  on  us  by  the  Office  of  Management  and  Budget.  If  you  heard  Mr.  Sitrin 
yesterday  you  know  that  indeed  they  are  finally  at  the  point  of  looking  at  better 
systems  for  dealing  with  estimating  inflation. 

Same  Inflation  Rate? 

One  of  those  is  the  use  of  more  detailed  rather  than  aggregate  measures.  One 
part  of  OMB's  problem  is  that  they  want  to  give  us  the  same  inflation  rate  as  any 
other  agency,  but  we  spend  about  98  percent  of  the  Federal  Government's  money 
on  fuel.  Since  fuel  has  been  inflating  very  rapidly,  it  is  quite  obvious  they  are  go¬ 
ing  to  have  to  treat  us  differently  in  that  area.  OSD  did  step  out  of  hound-  i.e., 
outside  the  guidance  given  by  OMB  this  year,  and  allowed  the  services  mc-e  for 
fuel  inflation.  By  doing  that  1  think  we  finally  got  OMB's  attention  to  the  need  to 
create  inflation  measures  which  break  down  to  commodity  level.  To  do  that  bet¬ 
ter,  we  need  better  ideas  as  to  what  tailored  price  indices  would  be  for  aircraft 
and  ships.  Now,  you  say  we  already  have  those,  but  the  one  we  use  for  ships,  the 
last  1  knew,  was  still  based  on  a  1950s  cargo  ship.  The  one  which  is  used  in  some 
areas  for  aircraft,  for  example,  by  the  Congressional  Budget  Office  is  also  based 
on  a  commercial  aircraft  of  considerable  age  and  doesn't  include  avionics.  We 
really  need  better  ideas  from  all  of  you  as  to  how  to  tailor  indices  for  specific 
weapon  systems.  We  need  to  have  better  historical  collection  of  inflation  as  well 
as  projections  for  the  budget  years.  The  Bureau  of  Economic  Analysis  (BEA)  has 
finally  put  together  an  effort  to  collect  historical  data,  but  there  are  some  dif¬ 
ficulties  with  that  effort.  It  needs  to  be  accepted  over  in  BEA  for  use  as  the  federal 
defense  deflator,  which  it  currently  is  not.  We  need  to  coordinate  throughout  the 
system  so  that  we  have  identical  approaches  or  at  least  close-to-identical 
approaches  to  measurement  and  planning  for  inflation  rather  than  one  measure 
being  used  by  CBO,  another  by  OMB,  another  by  OSD,  another  by  NAVAIR, 
NAVSEA,  Air  Force,  and  so  forth.  We  need  to  get  together  on  those,  and  I  intend 
to  see  that  that  happens.  And,  finally,  we  need  to  set  up  a  system  which  recon¬ 
ciles  our  estimates  of  inflation  with  the  actuals.  Those  are  the  challenges  I  see.  || 
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/4s  part  of  the  "Estimating  in  the  '80s"  general  session  meeting  at  the  Fifteenth 
Annual  Cost  Analysis  Symposium,  questions  were  taken  from  the  floor  and 
addressed  by  one  or  more  of  the  key  people  in  cost  analysis  assembled  to  par¬ 
ticipate  in  the  panel  discussion.  A  portion  of  the  transcript  from  that  session  is 
presented  below.  The  panelist  or  panelists  responding  to  each  question  are  iden¬ 
tified  by  name.  The  panel  was  moderated  by  Leroy  T.  Baseman,  Chairman  of  the 
Air  Force  Cost  Analysis  Improvement  Group.  Participants  on  the  panel  were 
Dr.  John  Morgan,  Direetor,  Cost  and  Economic  Analysis  Directorate,  Office  of 
the  Assistant  Secretary  of  Defense  (Program  Analysis  and  Evaluation); 
Mr.  Wayne  Allen,  Director  of  Cost  Analysis,  Office  of  the  Comptroller  of  the 
Army;  Mr.  Riner  Payne,  then  Deputy  for  Financial  Systems  and  Analysis,  Head¬ 
quarters  U.S.  Air  Force:  Mr.  Joseph  T.  Kammerer,  Director,  Cost  Analysis 
Division,  Headquarters  Naval  Material  Command;  and  Dr.  Michael  Sovereign, 
Director  of  Special  Projects,  Office  of  the  Assistant  Secretary  of  Defense 
(Comptroller^. 


Question;  I'm  pleased  that  we  are  talking  about  challenges  of  the  '80s.  I'm  sad¬ 
dened  that  we  missed  what  I  consider  the  most  significant,  that  being  a  time  to 
freeze  requirements.  We  all  know  that  what  drives  our  cost  estimates  is  not  better 
CERs,  or  better  data  bases,  or  better  mathematics,  or  better  communications.  In 
my  view  it  is  this  constant,  incessant,  undocumented,  creeping  problem  of 
changes  in  requirements.  So  I  would  throw  a  challenge  to  each  of  you  individual¬ 
ly  to  influence  and  to  do  what  you  can  to  at  least  put  a  pause  to  this  constant 
change  in  requirements  so  that  those  of  us  who  have  to  make  cost  estimates  can 
do  that  better  job  that  you  ask  us  to  do. 

Response  (Allen);  We  are  talking  about  a  systemic  problem.  It  is  a  problem  of 
how  do  we  obtain  increased  stability  in  the  acquisition  process,  or,  said  another 
way,  how  do  we  minimize  di.sruption?  That,  in  the  Army's  case,  gets  back  to 
something  as  fundamental  as  our  internal  requirements-setting  process,  how 
many  players  we  have  in  that  process,  and  how  many  times  those  players  are 
second-guessed  as  requirements  issues  pass  up  and  down  the  hierarchy.  The 
number  of  voices  that  must  be  accommodated  contributes  to  the  instebility  in  the 
requirements  over  time.  All  of  which  says,  I  don't  have  a  good  answer  to  your 
question.  I  have  noted  that,  within  the  Army,  we  are  talking  about  requirements 
stability  and  this  is  at  the  three-  and  four-star  level.  However,  institutionally  and 
systematically,  we  tend  to  shy  away  from  the  word  "freeze"  as  we  want  "flexibili¬ 
ty."  It  has  to  do  obviously  with  such  things  as  the  annual  funding  or  annual 
budgeting  process  and  how  that  process  causes  instabilities,  and  a  lot  of  that  is 
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beyond  our  control.  But  we  can't  carry  that  thought  too  far  or  it  can  become  a 
cop-out  for  not  trying  to  bring  stability.  There  are  some  mechanisms  that  we  are 
working  with  in  the  Department  of  the  Army  to  try  to  bring  about  increased 
stability.  One  such  mechanism  is  a  better  statement  of  materiel  system  re¬ 
quirements  from  the  outset  to  reduce  downstream  turbulence. 

Response  (Kammerer):  In  responding  to  your  comment,  Wayne,  it  seems  to 
me  that  we,  as  cost  estimators,  may  be  part  of  the  problem,  and  1  think  we  should 
take  part  of  the  blame.  After  all,  if  our  cost  estimates  were  not  continually  chang¬ 
ing,  the  decision-makers  would  not  have  to  change  their  quantity  requirements. 
When  one  weapon  system  constitutes,  say  25  percent  of  your  procurement 
dollars,  and  suddenly  increases  in  cost  estimates  raise  the  ratio  to  50  percent  of 
available  procurement  dollars,  something  must  be  cut  in  order  to  live  within  the 
constraints.  Cuts  in  quantities  raise  unit  production  prices  and  it's  this  sort  of 
dog-chasing-its-tail  syndrome  that  takes  place. 

Response  (Sovereign):  We  can't  simply  freeze  requirements;  we  all  know  that. 
What  we  need  to  do  is  better  understand  and  better  plan  for  those  requirements. 
The  design-to-cost  type  of  question,  the  affordability  question,  those  things  have 
to  be  integrated  in  the  bigger  picture  of  cost  estimating.  And,  if  you  can't  hang  on 
to  all  of  those,  you  don't  have  a  program  that's  going  to  be  stable,  and  1  think  it  is 
program  stability  rather  than  requirement  stability  that  we  need. 

Response  (Morgan):  I'm  not  sure  I  understand  exactly  what  you  mean  by 
"changing  requirements."  Are  you  talking  about  modification  programs,  the  con¬ 
ventional  ship  problem?  We  can't  get  a  ship  built  for  a  reasonable  cost  because 
there  are  frequent  revisions  that  change  the  configuration  of  that  ship,  for  exam¬ 
ple,  to  put  on  new  weapon  systems,  or  new  electronic  capabilities.  Or  are  you 
talking  in  a  larger  sense  about  the  characteristics  of  total  weapon  systems  as 
related  to  the  threat  that  faces  us  and  as  required  in  the  initial  acquisition 
specifications?  If  you  are  talking  about  the  latter,  it  seems  to  me  that  you  are  con¬ 
fronting  a  hopeless  task  if  you  think  we  are  going  to  forego  the  opportunity  to 
develop  a  better  new  weapon  system  capability  in  regard  to  the  threat.  Now,  if 
you  are  talking  about  the  former,  that  is  a  question  to  some  extent  of  discipline 
and  management  in  freezing  a  configuration  and  sticking  to  it.  Which  is  it  that 
you  mean? 

Questioner:  The  characteristics. 

John  Morgan:  In  other  words,  alternative  one,  the  constant  change  in  the  ship 
before  you  can  get  the  thing  built.  Well,  1  contend  that  you  are  speaking  about  a 
problem  that  is  beyond  the  capability  of  this  panel  to  address.  Our  only  hope  as 
cost  analysts  is  to  point  out  the  cost  implications  to  the  managers,  but  they  still 
will  make  the  final  decisions 

Response  (Payne):  Yes,  I  think  we  ought  to  take  into  account  Dr.  Perry's  pitch 
for  affordability  analysis.  While,  from  a  cost  analyst's  point  of  view,  life  would 
be  much  simpler  if  we  cut  out  the  turbulence,  we  have  to  recognize  that  we  need 
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to  enhance  characteristics  in  a  lot  of  cases  to  keep  up  with  the  threat.  If  the  system 
works  properly,  the  cost  analyst's  job  is  to  provide  good  alternative  costs  so  that 
the  DSARC  can  do  its  proper  job.  Obviously,  we  have  configuration  control 
boards  and  a  lot  of  things  like  that  which  presumably  tend  to  restrict  modifica¬ 
tions  in  major  systems  to  those  things  that  are  generally  agreed  to  as  essential.  I 
don't  think  there  is  any  realistic  expectation  of  freezing  changes  merely  to  accom¬ 
modate  the  costers,  nor  do  I  happen  to  think  it  desirable. 

Comment  from  the  floor;  I  think  what  the  gentleman  hit  on  was  the  real 
challenge  of  the  '80s.  We  are  not  going  to  settle  or  stabilize  the  financial  manage¬ 
ment  system  or  the  requirements  for  weapon  systems  or  any  of  these  other  things. 
What  our  real  challenge  is,  is  to  adapt  our  methodologies  and  our  techniques  to 
accommodate  these  so  that  we  can  do  our  job  even  better.  . 

Question:  With  regard  to  inflation,  which  seems  to  be  the  most  common 
denominator  judged  by  all  the  panel  members,  for  10  years  we  have  been  using 
OSD/OMB  inflation  factors.  And  we  keep  saying  that  the  factors  are  never  what 
they  should  be,  and  are  continually  critical  about  what  is  published.  I  just  wonder 
if  we  shouldn't  perhaps  be  more  concerned  with  exacting  nominal  factors  and  liv¬ 
ing  with  those  factors  for  inflation,  making  periodic  changes  as  opposed  to  issu¬ 
ing  OSD/OMB  factors  three  times  a  year,  which  causes  tremendous  aggravation 
and  gyrations  of  the  system.  It  is  not  so  much  the  mechanics  and  methodology  of 
how  to  determine  whether  it  should  be  the  GNP  deflator  or  some  nominal  factor 
of  S  to  10  percent,  but  establishing  a  procedure.  I  think  OSD  owes  us  leadership 
which  provides  for  a  more  stable,  uniform  application  and  treatment  of  how  the 
mechanism  will  exist.  Now,  with  regard  to  individual  systems  and  projects, 
because  no  nominal  value  would  ever  be  right,  as  no  published  GNP  deflator 
would  ever  be  right,  what  is  the  mechanism  that  would,  for  example,  provide  ex¬ 
tra  funds  when  the  XM-1  tank  actual  inflation  is  12.2  percent  and  the  GNP 
deflator  value  published  by  OSD  is  9.1  percent!  We  need  to  get  a  uniform  ap¬ 
proach.  We  simply  need  leadership  in  the  policy  area. 

Response  (Sovereign):  There  are  several  things  in  what  you  said.  The  last  item 
I  listed  was  a  system  for  reconciling  our  estimates  and  our  actuals,  and  that's 
probably  more  important  than  a  precise  estimate  of  what  inflation  is  going  to  be. 
1  totally  agree  with  you  and  I  think  we  are  going  to  move  to  do  something  there. 
Now,  your  other  point  was  the  three-times-a-year  adjustments.  There  again  we 
are  driven  by  the  President  trying  to  put  together  a  budget  in  the  January  time 
frame  to  send  to  Congress.  If  we  have  a  shorter  planning  process,  maybe  we 
could  reduce  that  from  three  to  two  times,  but  I  don't  think  we  are  going  to  be 
able  to  independently  step  out  and  send  over  a  budget  which  is  based  on  a 
nominal  rate  whereas  everybody  else  is  using  what  OMB  tells  them.  Does  that 
partially  answer  your  question! 

Question:  It  answers  it,  but  it  does  not  satisfy  me. 

Response  (Baseman):  I  would  like  to  make  a  comment  on  the  general  subject 
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of  inflation.  We  have  not  addressed  the  thing  that  goes  with  the  DOD  inflation 
rates  and  that  is  the  combination  of  the  inflation  rates  and  the  expenditure  pat¬ 
terns.  In  a  lot  of  cases  we  are  not  hurting  as  much  as  we  say  we  are  hurting. 

Response  (Kammerer):  In  my  presentation,  I  mentioned  the  outlay  rate  prob¬ 
lem.  It  seems  that  the  OSD  inflation  rates  are  always  low  in  the  out-years.  And 
yet,  it  is  these  inflation  rates  that  are  used  in  calculating  near-term  appropriation 
dollar  requirements  when  we  take  account  of  the  expenditure  profile.  These  rates, 
then,  are  very  important  in  calculating  the  dollars  necessary  to  execute  a  pro¬ 
gram:  We  must  do  a  better  job  of  forecasting  the  out-year  escalation  rates  with 
more  realism.  Remember,  we  are  not  recapturing  these  funds  that  we  lose.  OSD 
revises  the  rates  three  times  a  year  and  there  is  always  the  proclivity  to  forecast 
low  rates  in  the  out-years.  1  would  like  to  see  the  services  coming  forward  to 
OSD,  as  the  Navy  does  with  their  shipbuilding  indices,  saying  this  is  what  our 
research  has  shown,  this  is  what  we  think  the  economy  will  do  to  our  programs, 
and  we  would  like  to  have  OSD  review  and  approve  the  indices  we  have 
developed  for  our  programs.  I  believe  this  should  be  the  procedure.  If  OMB  is 
open  to  this  procedure,  and  we  can  convince  them,  then  we  should  institu¬ 
tionalize  the  process.  I  think  sitting  back  and  accepting  the  OMB/OSD  inflation 
indices  is  not  the  way  to  proceed.  We  must  step  forward  and  do  a  better  job 
ourselves  instead  of  just  complaining  that  the  rates  are  too  low. 

Response  (Baseman):  Joe,  1  have  to  interject  a  point:  Is  it  really  possible  to 
convince  anyone  of  what  actual  inflation  has  done  to  us,  unless  we  create  a 
system  for  measuring  the  impact  of  inflation  we  experienced?  I  don't  know  how  it 
is  in  the  Navy  and  the  Army,  but  quite  frankly  1  don't  think  we  in  the  Air  Force 
have  a  thorough  enough  system  for  determining  what  the  impact  of  inflation  has 
actually  been.  By  the  time  you  work  down  to  jobbers  and  sub-jobbers  and  fifth- 
tier  subcontractors,  we  can  only  generalize  as  to  what  we  have  actually  ex¬ 
perienced.  Until  you  can  do  that,  can  you  really  sell  something  different  from 
what  we  are  given? 

Response  (Kammerer):  Although  I  think  it's  a  difficult  problem  to  measure  the 
inflation  we  have  experienced  in  our  systems,  I  do  agree  that  we  need  to  do  it — it 
should  be  one  of  our  top  priorities.  I  think  this  is  what  Mike  Sovereign  has  been 
advocating. 

Question:  Does  the  government  ever  try  to  determine  the  effect  of  our 
publishing  our  planning  information?  1  can  think  of  two  situations:  One,  the 
Navy  has  the  fleet  modernization  program  wherein  we  list  the  planning  figures 
for  all  the  ship  alterations;  therefore,  contractors  know  what  the  customer  is  will¬ 
ing  to  pay  for  the  job.  In  the  case  of  inflation,  if  we  get  higher  rates  and  publicize 
this,  won't  industry  learn  of  them,  and  won't  it  have  the  effect  of  increasing  infla¬ 
tion? 

Response  (Morgan):  I'd  like  to  say  something,  if  1  may,  on  this  question  of  in¬ 
flation.  As  you  probably  know,  there  are  occasions  in  our  analytical  activities  in 
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OSD  when  we  take  into  consideration  what  we  call  progam-unique  inflation 
guidance  in  putting  together  the  POM  and  budgets.  In  providing  this  guidance, 
I'm  not  saying  the  rates  are  always  right.  Don’t  misunderstand  me;  I  think  we 
really  should  know  what  the  different  rates  are  going  to  be  by  different  resource 
areas.  But  I  don't  think  in  putting  together  a  POM  or  budget  that  you  can  always 
get  down  to  sufficient  detail  to  have  variable  rates  for  all  kinds  of  resource  areas 
of  weapon  systems.  Furthermore,  I'm  not  confident  that  the  decision-making  is 
going  to  be  influenced  dramatically  by  having  absolute  accuracy  and  ability  to 
forecast  inflation  rates.  I  think  decisions  are  made  on  different  allocation  bases 
than  that.  So  number  one,  program-unique  inflation  is  recognized  in  analyses 
that  we  do  as  we  are  moving  into  the  production  phase  on  weapon  systems  and 
we  have  the  services  provide  those  to  us.  Secondly,  1  don't  think  that  resolution 
of  the  inflation  rate  projection  2  years  before  you  are  going  to  start  the  full-scale 
engineering  development  of  a  weapon  system  is  going  to  be  a  major  influence  on 
the  decision. 

Response  (Payne):  I'd  like  to  add  to  that.  I  think,  and  I  say  this  al  the  risk  of 
being  drummed  out  of  the  fraternity,  if  you  look  at  the  iterative  nature  of  the 
budgeting  process,  an  understatement  of  inflation  at  a  given  point  in  time 
presumably  is  correctable  downst.'eam.  Now,  I  know  we  get  charged  with  cost 
growth,  and  that  is  galling  to  some  of  us.  But,  on  the  other  hand,  I  would  invite 
you  to  the  situation  we  would  have  if  we  went  to  individual,  program-unique  in¬ 
flation  rates.  We  tried  that  ii  the  Air  Force  for  awhile  before  OSD  standard  rates 
were  invoked,  and  I'm  not  si  re  from  the  standpoint  of  equitable  distribution  of 
forces  and  resources  that  program-unique  rates  add  any  more  to  the  process  than 
having  consistent  rates  applied  across  the  board.  Now,  I'm  not  satisfied  with  the 
rates  any  more  than  some  of  the  rest  of  you  are;  however,  I  think  there  is  a  virtue 
in  uniformity.  The  other  reason  that  I  guess  I'm  concerned  about  addressing  infla¬ 
tion  in  too  much  depth  is  that  this  is  an  area  where  we  can  fritter  away  literally 
thousands  of  man-hours  on  a  marginally  cost-effective  basis.  It's  a  popular  sub¬ 
ject  to  play  with,  but  it's  one  that  I  fear  has  no  real  agreed  solution.  So,  I'm  in 
favor  of  adopting  the  standard  rate  and  trying  to  improve  the  standard,  but  not 
encouraging  everybody  in  the  cost-estimating  business  to  strike  out  on  his  own  to 
decide  what  the  proper  rates  would  be. 

Question:  On  that  whole  issue  of  inflation,  I  wonder  if  there  isn't  an  alter¬ 
native  to  forecasting  inflation.  Why  not  take  inflation  out  of  our  budget 
altogether?  Is  there  a  possibility  of  considering  budgeting  in  constant  dollars  and 
adjusting  the  actually  appropriated  dollars? 

Response  (Allen):  The  closest  thing  to  that  I've  heard  about  came  in  the  course 
of  the  Brooks  testimony  on  weapon  system  cost  growth.  It  surfaced  as  the  discus¬ 
sion  of  inflationary  forces  showing  up  on  the  selected  acquisition  reports  (SARs). 
Congress  understands  the  cost  growth  on  weapon  systems  attributed  to  inflation 
and  there  was  a  comment  to  the  effect  that  maybe  all  of  the  SARs  ought  to  be 
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done  only  in  constant  dollars  so  that  the  Congress  could  clearly  differentiate  cost 
growth  attributable  to  inflation  and  cost  growth  attributable  to  non-macro 
economic  forces.  And,  from  time  to'  time  that  has  ebbed  and  flowed  in  terms  of 
how  to  present  that  information.  Currently,  as  you  know,  SARs  are  presented 
both  ways,  in  both  constant  and  current  dollars,  and  at  one  time  there  were  alter¬ 
native  inflation  rate  assumptions.  But  beyond  that  I  have  heard  no  proposal  to 
talk  of  cost  estimates  only  in  constant-dollar  terms.  Obviously,  you  can't  buy 
your  program  in  constant-dollar  terms.  You  can  only  buy  your  program  in 
current-dollar  terms,  so  we  have  to  deal  with  those  realities  in  the  budgeting  proc¬ 
ess. 

Question:  Regarding  the  challenges  of  the  '80s,  I  suggest  introducing  some 
technological  electronic  network  for  the  distribution  of  information  among  DOD 
sites  similar  to  those  for  educational  institutions.  We  find  a  network  down  in 
North  Carolina  between  educational  institutions  for  interchange  of  information 
programs.  In  Michigan,  we  find  a  network  where  universities  interchange  infor¬ 
mation,  techniques,  etc.  When  you  can  get  on  a  system  you  can  tap  a  different 
site  having  a  different  computer,  and  utilize  its  capabilities  and  its  data  base. 
What  I  wonder  about  in  the  electronic  age  of  the  '80s  as  we  get  more  and  more  ad¬ 
vanced  in  that  era,  is  why  we  don't  accelerate  the  interchange  of  information 
among  the  different  services  through  a  computer  network  for  rapid  transfer  of  in¬ 
formation.  I  just  throw  that  out  and  raise  the  question  of  whether  there  has  been  any 
thought  toward  that  or  should  there  be  any? 

Response  (Morgan):  Well,  we  have  thought  about  that  possibility  with  regard 
to  data  bases  for  different  weapon  systems.  We  would  have  in  OSD  a  weapon 
system  data  base  that  could  be  made  available  to  people  in  the  field  and  we  would 
have  a  link-up  with  the  services  so  we  ^ll  have  access  to  the  best  possible  data 
base  master  structure.  Everybody  could  tap  I  ita  base.  Now,  there  are  some 
problems  in  that.  First  ot  all,  one  problem  we  have  is  getting  money  for  studies  in 
order  to  get  something  like  this  put  together.  We  can't  even  get  enough  money  to 
get  our  own  data  on  a  computer  in  OSD,  and  our  study  money  has  just  been 
zeroed  out  by  the  Congress.  We  hope  to  get  that  back,  but  anyway,  this  is  one 
problem — getting  the  dollar  resources  to  be  able  to  put  something  like  that 
together.  Second  is  the  question  of  availability.  Is  the  Army  going  to  let  us  have 
all  of  their  data  base  down  at  the  Missile  Command  on  missiles?  Is  the  Air  Force 
going  to  open  up  all  of  the  Aeronautical  Systems  Division's  data  bases  to  OSD? 
These  are  some  of  the  political  questions.  So,  what  you  have  described  is  an  op¬ 
timum  situation;  getting  there  is  a  problem. 

QueslioniThe  last  couple  of  days  we  have  been  hearing  the  theme  of  profes¬ 
sionalism.  What  I  would  like  to  ask  of  the  panel  members  is,  is  it  time  now  for 
this  discipline  to  initiate  an  American  Society  for  Cost  Analysis,  publish  a  jour¬ 
nal,  and  have  dues-paying  members? 


Estimating  in  the  '80s 


47 


Response  (Allen):  Well,  I  think  in  some  respects  we  are  moving  along  that  line 
now.  First  of  all,  let  me  agree  with  Riner  Payne — no  one  wants  a  cult.  We  are  not 
interested  in  that  kind  of  approach  or  thought  process,  but  it  does  seem  ap¬ 
propriate  to  talk  in  terms  of  a  formal  professional  society.  With  that  thought  in 
mind,  I  suggest  you  may  want  to  consider  the  National  Estimating  Society  (NES) 
which  is  on  the  national  level  and  has  been  in  business  now  for  2  years.  This  par¬ 
ticular  society,  the  National  Estimating  Society,  is  principally  made  up  of 
aerospace  concerns.  I'd  say  the  mix  in  terms  of  membership  is  something  like  90 
percent  industry  and  10  percent  government.  But  it  does  have  the  basic  beginning 
of  the  type  of  thing  you  are  getting  at.  There  is  the  American  Association  of  Cost 
Engineers,  I  believe,  the  specifics  of  which  I'm  not  familiar  with.  When  you  stand 
back  from  this  evolving  process,  the  answer  to  your  question  is,  yes,  I  feel  quite 
sincerely  we  are  ready  as  a  profession  and  as  a  community  to  move  along  those 
lines.  II 


Alternative  Techniques 
for  Use  in  Parametric 
Cost  Analysis 

W.  Eugene  Waller 
Thomas  J.  Dwyer 

Cost  analysts  are  plagued  with  many  difficulties  in  trying  to  develop  a  cost 
estimating  model  or  a  point  estimate  for  military  systems.  One  of  these  dif¬ 
ficulties  is  the  limitations  in  the  statistical  procedures  available  to  the  practicing 
cost  analyst.  Another  is  the  difficulty  in  developing  cost  models  that  are  not  made 
obsolete  by  the  rapid  advance  of  technology  in  military  equipment.  This  paper 
describes  (1)  some  alternatives  to  the  conventional  statistical  methods  of  develop¬ 
ing  cost  estimating  models,  and  (2)  a  means  for  normalizing  cost  estimating 
models  for  technology  advance  so  that  these  models  remain  relevant  even  though 
the  technology  is  evolving  rapidly.  First,  however,  it  is  necessary  to  briefly 
describe  the  conventional  method  of  cost  analysis  for  those  unfamiliar  with  this 
discipline. 

Parametric  Cost  Analysis 

The  cost  analysis  discipline,  as  considered  here,  typically  involves  the 
development  and  use  of  parametric  cost  models  or  cost  estimating  relationships 
(CERs).  These  parametric  models  usually  result  from  the  application  of  multiple 
linear  regression  techniques  to  a  sample  of  cost  and  technical  information.  The 
CER  resulting  from  this  process  relates  the  dependent  variable  cost  to  technical 
performance  variables  of  the  system  or  other  variables  which  impact  cost. 

Multiple  linear  regression  allows  one  to  fit  a  CER  of  the  following  form: 

Cost  ^ 

Where  the  X's  are  variables  which  impact  cost,  the  b's  are  the  fitted  regression 
coefficients,  and  E  is  an  error  (uncertainty)  term.  For  example,  a  CER  for  the  pro¬ 
duction  cost  of  an  RF  transmitter  might  include  average  power  and  duty  cycle  as 
explanatory  variables.  There  can  be  some  flexibility  in  the  specification  of  a  CER. 
A  CER  of  the  following  form  may  be  fit  using  multiple  linear  regression; 

Cost  =  ...  E 
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This  is  achieved  by  a  logarithmic  transformation  of  the  dependent  variable  cost 
and  the  independent  variables  X  and  then  application  of  the  standard  multiple 
linear  regression  technique. 

The  use  of  multiple  linear  regression  produces  various  fit  statistics  such  as  R?-, 
standard  error  and  t-statistics,  which  measure  the  goodness  of  fit  and  the  uncer¬ 
tainty  of  the  CER.  These  measures  are  often  used  in  selecting  the  best  CER  from 
alternative  CERs  and  in  assessing  the  accuracy  of  the  CER. 

Parametric  cost  estimates  are  used  in  the  early  stages  of  a  program,  usually  in 
concept  formulation  or  in  the  research,  development,  test  and  evaluation 
(RDT&E)  phase.  This  type  of  estimate  is  replaced  later  in  RDT&E  and  production 
by  projections  of  cost  experienced  in  the  program.  Parametric  estimates  are  only 
approximations;  when  there  are  only  a  few  relevant  programs,  limiting  the  sam¬ 
ple  size,  the  number  of  variables  that  can  be  incorporated,  in  turn,  are  limited. 

Model  Selection  Criteria 

Frequently  in  cost  analysis  an  analyst  is  faced  with  selecting  among  alter¬ 
native  specifications  of  cost  models.  How  does  one  select  the  best  model?  All  too 
often  the  criterion  used  for  the  goodness  of  the  model  is  the  value.  The  R^ 
value,  however,  only  describes  how  well  the  model  explains  variations  in  the  data 
about  the  cost  mean.  A  better  measure  of  the  goodness  of  fit  in  our  view  is  the 
standard  error  of  the  model.’  It  is  an  important  component  (along  with  the 
variance  of  b  coefficients)  of  the  prediction  interval.  Minimizing  the  prediction 
interval  is  a  primary  concern  of  the  cost  analyst;  consequently,  achieving  a 
minimum  standard  error  should  be  a  primary  goal. 

Minimizing  the  standard  error  and  maximizing  may  lead  to  the  same 
model.  High  R^  values  can  be  deceiving,  however,  because  substantial  error  can 
still  exist  in  the  model  as  shown  in  the  following  example.  Figure  1  exhibits  a 
situation  which  sometimes  arises  in  cost  modeling.  Cost  data  are  available  for  a 
given  type  of  hardware,  e.g.,  rocket  motors  spanning  a  very  wide  range  of  per¬ 
formance.  It  is  also  known  that  the  performance  variable  plotted  in  the  figure  is  a 
cost  driver.  When  all  of  the  data  are  included  in  the  sample,  a  deceivingly  high  R^ 
is  achieved  in  a  single  variable  linear  regression  of  cost  against  the  performance 
variable  because  of  the  wide  spread  in  cost  values  about  the  mean.  If  the  cost  data 
spans  three  orders  of  magnitude  (leading  to  a  high  total  sum  of  squares  about  the 
mean),  an  R^  of  95  percent  for  the  single  model  and  a  sample  size  as  shown  in 
Figure  1  will  produce  a  standard  error  of  about  30  percent,  a  significant 


1.  The  standard  error  is  the  square  root  of  the  sum  of  squared  residuals  divided  by  the  degrees  of 
freedom.  The  square  of  the  standard  error  is  the  best  linear  unbiased  estimator  of  the  variance  of  the 
error  term  of  the  model. 
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FIGURE  1 

r2  and  Standard  Error  Comparison 


estimating  error.  The  figure  reveals  that  a  large  proportion  of  the  variation  about 
the  mean  cost  is  explained  by  the  single  variable  regression.  It  is  apparent, 
however,  that  two  different  cost  functions  apply  and  that  the  standard  error 
(variation  about  the  line)  could  be  reduced  substantially  by  stratifying  the  data 
into  two  samples  and  fitting  each  one  separately  as  shown  in  the  lines  labeled 
"Model  A"  and  Model  B"  in  the  figure.  It  is  possible  for  the  values  of  the 
stratified  models  to  be  less  than  the  value  for  the  single  model  even  though  the 
stratified  model  is  a  superior  model.  Admittedly,  this  is  an  extreme  case,  but  just 
such  a  problem  arose  in  a  recent  study  where  CERs  were  being  developed  for  pro¬ 
pulsion  systems  for  strategic  and  tactical  missiles.  There,  in  a  model  of  cost  as  a 
function  of  total  impulse,  it  was  necessary  to  stratify  the  sample  by  mission  to 
produce  the  best  model. 
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Another  example  of  the  use  of  the  standard  error  in  selecting  a  best  model 
occurred  in  an  evaluation  of  two  different  statistical  approaches  to  measuring 
state-of-the-art  advance  (SOAA).  The  SOAA  measure  was  included  in  an 
RDT&E  cost  estimating  model  for  aircraft  turbine  engines  as  an  independent 
variable,  the  cost  hypothesis  being  that  SOAA  was  a  substantial  determinant  of 
RDT&E  cost.  The  two  different  methods  for  measuring  SOAA  were  evaluated 
based  on  (1)  the  RDT&E  cost  estimating  model  goodness  of  fit,  (2)  historical 
simulation  results,^  and  (3)  how  well  they  estimated  two  programs  that  were  not 
part  of  the  data  base  used  to  quantify  the  models.  The  goodness  of  fit  criterion 
used  was  the  standard  error  of  the  two  models.  There  was  a  considerable  dif¬ 
ference  between  each  model's  standard  error  (.11  vs.  .14)  even  though  the 
values  were  essentially  identical  (.970  vs.  .972).  The  historical  simulation  results 
clearly  confirmed  that  the  model  with  the  smallest  standard  error  was  the  best 
model,  as  did  a  later  exercise  of  estimating  the  two  programs  excluded  from  the 
data  base.^ 

Non-Linear  Fitting  Techniques 

Often  in  cost  analysis,  one  develops  a  hypothesis  about  cost  that  cannot  be  fit 
to  the  data  by  conventional  multiple  linear  regression  techniques,  e.g.. 
Cost  =  -t-  +  £•  Usually  the  form  of  the  cost  function  is  then 

altered  so  that  it  can  be  fit.  This  is  a  second-best  solution.  A  related  problem  is 

that  a  CER  consisting  of  a  sum  of  terms  (e.g.,  bp  +  hjXj  -i-  fj2^2^ 

conventionally  where  a  multiplicative  log  normal  error  term  is  assumed.  If  fit 
assuming  a  linear  error  term,  large  cost  programs  dominate  the  result  such  that 
residuals  (in  percent  of  total  cost  terms)  tend  to  be  smaller  for  these  programs. 
This  is  contrary  to  our  perception  of  cost  uncertainty  and  so  it  is  desirable  to  have 
a  means  of  fitting  such  a  function  with  a  log  normal  error  term. 

Two  computer  programs  have  been  developed  to  handle  these  problems. 
While  non-linear  regression  packages  are  generally  available  and  could  be  used, 
these  two  programs  are  custom  designed  for  particular  needs.  The  first  regression 
package  (LOGFIT)  allows  only  the  limited  non-linear  capacity  to  fit  cost  as  a 
linear  function  under  the  assumption  of  a  multiplicative  log  normal  error  term. 
This  mathematical  form  is  frequently  most  appropriate  in  cost  analysis.  This 
package  will  also  perform  standard  multiple  linear  regressions  (including 


2.  Historical  simulation  involves  removing  the  most  recent  program  from  the  sample,  refitting  the 
selected  model  form,  and  using  the  refitted  model  to  estimate  the  deleted  program. 

3.  C.  A.  Graver.  Historical  Simulation:  A  Procedure  for  the  Evaluation  of  ^timating  Procedures. 
General  Research  Corporation  CR-0364-1.  June  1969. 
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weighted  regressions)  and  it  also  allows  the  incorporation  of  prior  information 
(Bayesian  statistics). 

When  other  non-linear  forms  are  required,  a  more  general  non-linear  regres¬ 
sion  package  is  needed.  This  program  (CRVFIT)  is  capable  of  fitting  any  smooth 
non-linear  '  action  using  the  linearization  algorithm. 

Many  pportunities  have  arisen  in  our  experience  to  use  the  LCXjFIT 
algorithm  to  fit  a  sum  of  terms  cost  expression  with  a  log  normal  error.  One  ex¬ 
ample  of  this  occurred  in  a  study  of  military  software  development  costs.  Soft¬ 
ware  development  costs  are  related  to  the  number  of  machine  words  of  storage 
developed  for  the  system.  It  is  also  hypothesized  that  the  cost  per  word  varies 
with  the  type  of  software  developed.  In  the  study,  "operational"  (applications) 
software  words  were  segregated  from  "development  support"  words.  This  latter 
category  includes  all  non-operational  software,  i.e.,  simulation  software,  data  ex¬ 
traction  software,  etc.  Within  the  "operational"  words  category,  "test"  words 
were  segregated  from  all  other  words.*  The  hypothesis  to  be  tested  was  then: 


Cost 

where 

Cost 

^2 

^3 

bi,b2  &  ^3 

e 


(biXi  +  ^2^2  ^3^3)  ^ 

Software  Development  Cost  (Total) 
Operational  Words  Other  than  Test 
Operational  Test  Words 
Development  Support  Words 
Model  Coefficients 

Error  Term  (Log-normal  Distribution) 


The  coefficients  that  result  from  fitting  this  model  are  shown  in  Table  I  and 
are  contrasted  with  the  coefficients  for  the  same  model  fit  employing  a  linear 
error  term.  There  is  a  significant  difference  between  these  two  models,  especially 
in  the  coefficients  for  the  test  and  development  support  words. 


TABLE  1 

Software  CER  Coefficients 

*>2 

*>3 

LOG  NORMAL  ERROR  COEFFICIENTS 

71 

17 

21 

LINEAR  ERROR  COEFFICIENTS 

56 

10 

40 

4.  Test  words  are  words  used  for  performance  monitoring/fault  location  (PM/FL). 
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The  software  study  also  illustrates  the  use  of  the  non-linear  algorithm 
CRVFIT.  This  algorithm  was  applied  to  the  following  CER  form; 

Cost  =  biXi^^  +b^X2  +  ^5  X3  -h  e 

where  all  variables  are  defined  as  they  were  previously  and  six  coefficients  (b's) 
are  fit.^  The  resulting  coefficients  are  shown  in  Table  II.  One  expects  to  see  ex¬ 
ponents  closer  to  one,  descriptive  of  linear  cost  behavior  with  words.  This  dif¬ 
ficulty  in  interpretation  led  to  the  rejection  of  this  model. 

Bayesian  Approach 

In  conventional  parametric  analysis,  a  homogeneous  set  of  data  is  processed 
to  establish  the  coefficients  of  the  CERs.  No  prior  knowledge  of  the  CER  is  assumed 
in  the  classical  least  squares  regression  approach.  However,  the  analyst  may 
know,  as  the  result  of  a  previous  study,  the  behavior  and  variance  of  the  cost  of 
an  item,  for  example  the  planar  array  portion  of  a  tracking  antenna,  yet  have 
costs  only  for  the  complete  tracking  antenna  including  the  gimbal  system.  This 
information,  under  a  Bayesian  approach,  can  be  used  to  advantage  where  one  is 
fitting  a  CER  that  consists  of  the  sum  of  two  terms,  one  term  representing  the 
planar  array  portion  and  the  other  the  gimbal.  The  prior  information  will  in¬ 
fluence  the  coefficient  on  the  array  variable  to  be  consistent  with  the  prior  data 
allowing  better  definition  of  the  coefficient  on  the  gimbal  variable. 

A  regression  program  has  been  developed  (tailored  specifically  to  parametric 
cost  analysis)  which  allows  the  incorporation  of  prior  information  into  the  CER 
development  process."  This  computer  program  allows  the  user  to  specify  prior  in¬ 
formation  about  CER  coefficients  in  terms  of  a  prior  expected  value  and  a  prior 
variance.  The  computer  program  then  produces  a  least  squares  estimate  by  com¬ 
bining  the  prior  information  with  the  regression  data  set  using  Bayesian  methods. 


TABLE  II 

Software  CER  Coefficients  (CRVFIT) 


A 

*>2 

*>3 

*>4 

A 

A. 

CRVFIT  COEFFICIENTS  60 

.97 

38.6 

.56 

2.9 

1.38 

5.  This  model  is  primarily  shown  for  illumination  since  there  are  only  12  programs  in  the  sample. 

6.  T.  J.  Dwyer,  Bayesian  Methods— The  Use  of  Prior  Knowledge  in  ParamefnV  Cosf  /Ana/ysis, 
Tecolote  Research,  Inc.  TM-70,  March  1977. 
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As  a  special  case  of  the  above,  the  analyst  can  specify  a  prior  variance  of  zero, 
effectively  fixing  the  coefficient  to  the  prior  expected  value.  This  can  be  a  useful 
technique,  especially  with  the  small  data  sets  which  occur  frequently  in  cost 
analysis. 

An  example  of  the  application  of  the  Bayesian  approach  to  the  development 
of  a  cost  estimating  relationship  for  radar  tooling  and  test  equipment  is  shown  in 
Table  III.  A  CER  of  the  form  bg  Xj  X2  £  had  been  developed  for  missile  tool¬ 
ing  and  test  equipment  cost  in  a  prior  study.  The  variable  Xj  was  the  cumulative 
average  cost  for  the  missile  and  the  variable  X2  was  the  peak  production  rate.^ 
The  value  for  the  constant  {>2  determined  by  the  fitting  process  was  .5.  We  had 
some  confidence  that  the  resulting  CER  was  a  good  one.  We  believed  that  this 
constant  would  apply  for  radar  tooling  and  test  equipment  as  well.  It  is  probable, 
for  example,  that  a  doubling  of  the  peak  production  rate  would  not  double  the 
tooling  and  test  equipment  complement  because  of  the  opportunity  for  sharing 
some  of  the  initially  purchased  equipment  by  the  second  production  line.  This  ra¬ 
tionale  led  to  the  incorporation,  in  the  fitting  process,  of  the  prior  missile  study 
information  on  the  expected  value  and  the  standard  deviation  of  the  coefficient 
b2.‘  Regression  coefficients  and  statistics  for  this  fitting,  together  with  the  same 
statistics  for  the  run  without  the  prior  information  is  shown  in  Table  111.  In  this 


TABLE  III 

Comparison  of  CER  Results  With  and  Without  Use  of  Prior  Information 


Coefficient 

t-Value 

Prior  Mean 

Prior 

Standard 

Deviation 

1.580 

5.97 

00 

.482 

2.78 

MODEL  2 

In  bo 

1.568 

bl 

0.00 

b2 

.493 

_ 

.500 

.140 

7.  hg  ,  bj  and  b2  were  constants. 

8.  This  information  is  produced  in  every  regression  run. 
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particular  instance,  the  effect  of  the  constraint  is  quite  small,  adjusting  the  bg  and 
bj  values  only  slightly.  Either  model  could  be  employed.  The  higher  t-value  on 
^2/  however,  reflects  the  greater  certainty  that  {>2  because  of  the  prior 

data.  It  also  yields  a  narrower  prediction  interval  for  a  point  estimate.  In  general, 
more  dramatic  results  than  shown  here  can  be  expected. 

This  example  also  shows  another  application  of  the  Bayesian  technique.  The 
value  of  hi  has  been  limited  to  1.0  without  deviation,  i.e.,  the  dependent  variable 
is  a  linear  function  of  Xj,  cumulative  average  cost.  This  is  a  common  requirement 
in  cost  analysis  that  can  be  handled  conventionally  by  transforming  the  depend¬ 
ent  variable;  however,  employing  the  Bayesian  approach  facilitates  the  process 
and  also  provides  fit  statistics  for  the  variable  of  interest,  cost,  not  some 
transformed  variable  such  as  the  ratio  of  tooling  and  test  equipment  cost  to 
cumulative  average  cost. 

Fitting  Learning  Curves 

While  learning  curves  may  be  the  most  discussed  subject  in  cost  analysis, 
there  are  a  few  subtle  difficulties  that  deserve  attention.  The  difficulties  are  not 
with  learning  curve  theory  itself,  but  with  the  problems  involved  in  actually 
fitting  a  learning  curve  to  historical  data. 

There  are  a  variety  of  learning  curve  theories,  all  of  which  state  that  unit 
manufacturing  costs  drop  as  more  units  are  produced.  The  two  most  popular 
formulations  are: 

1.  i/C  = 

where  i/C  is  the  cost  of  the  Qth  unit,  and  A  and  b  are  constants. 

2.  CAC  =  /IQ*’ 

where  CAC  is  the  cumulative  average  cost  of  the  first  Q  units,  i.e.,  CAC  is 
the  total  cost  of  the  first  Q  units  divided  by  Q  and  A  and  b  are  constants. 

The  constant  A  is  usually  termed  the  Theoretical  First  Unit  Cost  and  the  con¬ 
stant  b  is  the  slope,  although  the  slope  is  usually  given  in  percent  (100  x  2*). 

The  practical  difficulty  with  equations  1  and  2  is  that,  statistically,  the  model 
has  not  been  completely  defined  because  these  equations  lack  specification  of  the 
error  term.  When  completely  specified,  these  two  equations  can  produce  a  vari¬ 
ety  of  models,  four  of  which  are  given  below. 

l.a  UC  -  AQ^  ' 

where  US  is  the  co»i  Qth  unit. 

1. b  LAC  -  AN‘>  t 

where  LAC  is  the  lot  average  cost  and  N  is  the  unit  theory  lot  midpoint. 

2. a  CAC  ^  ^Q*’  e 

where  CAC  is  the  cumulative  average  cost  of  the  first  Q  units. 

2.b  LAC  -  AM'’  t 

where  LAC  is  the  lot  average  cost  and  M  is  the  synthetic  cumulative 
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average  theory  lot  plot  point,”  and  in  all  the  equations  E  is  a  lognormal 
error  term. 

Of  these  above  equations,  equations  l.a  and  l.b  are  unit  theory  equations 
corresponding  to  equation  2,  and  equations  2. a  and  2.b  are  cumulative  average 
theory  equations.  Notice  the  differences  in  the  dependent  variables  among  these 
equations. 

Statistically,  all  these  equations  are  distinct,  although  in  the  case  of  perfect 
data  (i.e.,  no  noise)  equations  1  will  agree  and  equations  2  will  agree.  Further¬ 
more,  there  can  be  no  statistical  comparison  between  how  well  the  unit  theory 
equation  l.a  fits  the  data  as  opposed  to  the  cumulative  theory  2. a  because  the 
dependent  variables  in  these  equations  are  so  radically  different.  There  can, 
however,  be  a  goodness  of  fit  comparison  between  equations  l.b  and  2.b. 

The  differences  between  the  equations  can  be  stated  non-technically  in  terms 
of  the  implied  weighting  each  equation  gives  to  each  data  point.  Equation  l.a 
gives  equal  weight  to  the  cost  of  each  unit.  However,  this  equation  can  seldom  be 
used  in  practice  because  most  military  equipments  are  bought  in  lots  of  a  quantity 
greater  than  one,  and  hence  the  cost  of  individual  units  is  not  available.  In  this 
case,  the  unit  theory  analyst  usually  turns  to  equation  l.b  where  each  lot  is  given 
equal  weight.  But,  in  comparing  l.a  to  l.b,  individual  units  which  belong  to 
larger  lots  are  given  less  weight  than  would  be  implied  by  equation  l.a. 

Equation  2. a  (which  is  the  usual  cumulative  form  used  in  practice)  has  the 
most  pronounced  weighting  among  the  four  equations  since  the  dependent 
variable  (CAC)  is  calculated  as  the  quantity-weighted  average  of  successive  lots. 
This  implicitly  weights  early  lots  more  heavily  than  later  lots.  Like  equation  l.a, 
this  equation  is  also  difficult  to  use  in  practice  because  its  use  requires  that  the 
cost  of  each  successive  lot  be  known.  For  example,  if  the  cost  of  the  first  lot  is 
unknown  (or  perhaps  contaminated  in  some  way)  this  equation  cannot  be  used 
even  if  a  good  history  is  available  for  succeeding  lots.  However,  it  can  generally 
be  used  more  often  than  equation  l.a. 

Equation  2.b  is  similar  to  equation  l.b  and  hence  the  weighting  is  the  same, 
allowing  a  statistical  comparison  between  l.b  and  2.b. 

To  cope  with  these  choices,  a  computer  program  has  been  developed  which 
by  iteration  can  fit  all  four  model  choices  to  the  data  (subject  to  the  restriction  on 
equations  l.a  and  2. a).  This  forces  a  comparison  of  the  four  equations  although 
we  typically  use  either  l.b  or  2.b  which  generally  give  similar  results. 

An  example  of  this  comparison  for  a  missile  seeker  is  given  below. Notice 
that  the  results  are  quite  different  for  equations  2. a  and  2.b  although  they  are 
both  based  on  the  same  equation,  differing  only  in  how  the  equation  is  fitted  to 
the  data. 

9.  The  point  at  ivhich  fhe  average  cost  is  equal  to  the  cumulative  average  cost. 

10.  Equation  l.a  could  not  be  fitted  because  lot  sizes  were  greater  than  one. 
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MISSILE  SEEKER 

A 

Slope 

Standard 

Error 

EQUATION  l.b 
(Unit  Theory) 

11.0 

95.5% 

.18 

EQUATION  2.a 
(Cumulative  Theory) 

17.8 

92.3% 

.04 

EQUATION  2.b 

11.8 

95.5% 

.18 

(Cumulative  Theory) 

As  the  example  shows,  the  different  equations  can  produce  very  different 
results.  Notice  that  equation  2. a,  by  averaging  lot  costs,  has  smoothed  most  of 
the  variation  out  of  its  dependent  variable  as  is  reflected  in  the  standard  error." 
In  fact,  the  use  of  equation  2. a  has  served  to  obscure  cost  trends  in  the  most  re¬ 
cent  lots.  In  this  example,  the  learning  curve  in  later  lots  began  to  flatten  out. 
Because  equation  2. a  weights  early  lots  most  heavily,  this  flattening  was  ignored. 
The  consequence  of  this  is  that  future  lots  would  probably  be  seriously 
underestimated  by  equation  2. a,  while  the  other  equations,  which  are  responding 
to  the  changing  trend,  would  produce  good  estimates. 

Technology  Advance 

Technology  advance  has  always  troubled  military  systems  cost  analysts. 
Creation  of  a  large  enough  sample  for  CER  development  usually  involves  in¬ 
cluding  equipment  that  was  introduced  up  to  10  or  more  years  ago.  Most 
technologists  in  the  electronics  industry  consider  that  era  to  be  ancient  history 
and  irrelevant  to  current  technology.  But  there  is  value  to  that  old  information.  It 
may  help  in  quantifying  the  underlying  cost  function  if  the  effect  of  technology 
advance  can  be  removed,  i.e.,  if  the  data  can  be  normalized  for  the  advance. 

One  method  that  has  been  used  to  accomplish  this  is  to  incorporate,  in  a  log¬ 
normal  linear  regression,  a  term  {b„X)  called  a  "technology  index,"  where  X  is  a 
calendar  year  variable  corresponding  to  the  year  of  initial  manufacture  of  the 
particular  item,  and  b„  is  the  regression  coefficient.  This  term,  assuming  h„  is 
negative,  acts  to  adjust  the  resulting  CER  downward  at  a  continuous  rate  year-to- 
year  thus  capturing,  in  an  approximate  way,  the  continuing  advance  of 
technology.  This  specification  of  the  cost  function  hypothesizes  that  technology 
advance  proceeds  at  a  uniform  rate  over  a  long  period  of  time.  One  may  argue 
that  technology  advance  is  discontinuous,  occurring  in  large,  discrete  jumps; 


11.  Note  that  the  standard  errors  are  not  comparable,  and  hence  the  standard  error  does  not  in¬ 
dicate  that  equation  2. a  is  superior. 
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however,  new  technology  is  usually  implemented  in  military  hardware  in  a  much 
more  gradual  way.  Take,  for  example,  solid  state  RF  amplifiers.  Although  these 
have  been  available  for  a  long  period  of  time,  final  power  amplifiers  for  virtually 
all  radars  are  electron  beam  tubes.  There  has  been  some  penetration  by  solid  state 
amplification  at  lower  power  levels,  e.g.,  as  drivers  for  final  amplifiers  or  for 
amplification  at  lower  frequencies,  but  this  has  been  a  gradual  process. 

An  example  of  the  incorporation  of  a  technology  index  in  a  CER  is  shown  in 
Figure  2  for  core  and  magnetic  film  memory.  Cost  per  bit  are  plotted  vs.  year  first 
manufactured.  The  steep  downward  slope  of  this  cost  function,  with  fiscal  year, 
is  the  result  of  the  technology  index  adjustment.  The  technology  index  shown 
reflects  a  17-percent-per-year  reduction  in  cost  per  bit.  Technology  indices  like 


FIGURE  2 

CER  for  Core  and  Film  Memory 
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these  have  been  established  in  other  areas  including  SAW  devices,  A/D  con¬ 
verters,  and  signal  processing. 

The  use  of  extraneous  or  prior  information  on  technology  advance  would 
seem  to  be  an  ideal  application  of  the  Bayesian  approach.  A  number  of  functions 
similar  to  Figure  2  have  appeared  in  the  digital  electronics  literature  presenting  a 
possible  source  of  prior  information  on  technology  advance  that  could  be  incor¬ 
porated  with  the  sample  data. 

Conclusion 

This  paper  has  described  some  of  the  methods  helpful  in  analyzing  cost  data 
for  military  systems.  These  methods  alleviate  a  number  of  the  difficulties  an 
analyst  is  faced  with  when  estimating  costs.  The  Bayesian  approach  allows  the 
analyst  to  benefit  from  prior  information  on  cost  behavior.  Non-linear  methods 
give  the  analyst  a  wider  and  often  more  appropriate  set  of  mathematical  forms  to 
choose  from  when  developing  CERs.  The  incorporation  of  a  technology  index  in 
a  CER  is  a  means  for  compensating  for  technology  advance  and  its  adverse  im¬ 
pact  on  CERs.  Finally,  alternate  means  of  fitting  learning  curves  provide  the 
analyst  with  increased  flexibility  and  better  visibility  in  the  assessment  of  the  true 
behavior  of  cost  with  quantity.  The  proper  use  of  these  tools  should  lead  to  better 
CERs  permitting  the  analyst  to  focus  his  attention  on  some  of  the  other 
formidable  problems  of  cost  analysis.  || 
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Major  Donald  E.  Crawford,  USAF  || 

In  1976  the  MX  missile  program  office  designed  a  computerized  cost 
model  that  recognized  that;  (1)  MX  estimates  would  be  based  on  other  solid-fuel 
missiles,  but  historical  data  would  be  limited;  (2)  MX  would  be  one  of  the  largest 
procurements  ever  attempted  by  the  U.S.  Air  Force;  (3)  the  cost  data  would  be 
subjected  to  intense  scrutiny  from  many  sectors;  and  (4)  numerous  changes 
would  occur  during  the  procurement,  requiring  immediate  repricing  of  the 
estimates. 

1  will  attempt  to  provide  a  background  and  a  general  description  of  the  com¬ 
puter  model  developed.  My  description  is  divided  into  major  components  for 
ease  of  understanding;  however,  all  applicable  costs  are  calculated  and  produced 
in  one  computer  product,  providing  a  true  life-cycle  cost  model  sensitive  to 
changes  in  concept  that  affect  either  acquisition  or  operation  and  support. 

The  concept  for  a  Minuteman  follow-on  missile  began  officially  with  a  pro¬ 
gram  management  decision  dated  June  11,  1973,  which  directed  Defense  Systems 
Acquisition  Review  Council  (DSARC)  I  for  January  1976.  That  DSARC  I 
milestone  presented  the  following  basing  possibilities:  vertical  shelter,  buried 
trench,  air  mobile,  and  Minuteman-silo  basing.  The  number  of  missiles  to  bt 
placed  on .  in  each  of  these  systems  varied  significantly;  therefore,  the  costs  of 
the  differer  otions  could  not  be  directly  compared. 

For  DSA  '  II,  the  program  office  was  directed  to  concentrate  on  ground- 
based  option  >nly.  The  first  DSARC  11  briefings  in  late  1978  presented  the 
following  optii  iS;  MX  in  vertical  shelter,  MX  in  horizontal  shelter,  and  MX  in 
buried  trench.  The  two  missiles  proposed  were  92  inches  and  83  inches  in 
diameter.  Some  basing  models  included  installation  of  missiles  in  existing 
Minuteman  silos  with  later  moves  scheduled  to  Southwest  bases.  As  a  result  of 
this  series  of  briefings,  the  program  office  was  directed  to  concentrate  its  efforts 
on  evaluating  the  air  mobile  MX.  There  was  no  aircraft  specifically  designed  to 
perform  this  mission;  therefore,  modifications  of  the  AMST,  Boeing  747,  and 
C-5,  with  seven  missiles  of  varying  weights,  were  considered  and  estimated. 
These  studies  were  completed  in  March  1979  and,  although  the  92-inch-diameter 
missile  was  selected,  the  available  system  was  considered  too  highly  technical  and 
costly  to  deploy. 

In  early  1979  another  study  was  made  of  the  basing  mode  with  emphasis  on 
those  requirements  that  would  result  from  congressional  approval  of  the  SALT  II 
treaty.  The  horizontal  shelter  and  buried  trench  were  emphasized  owing  to  the 
difficulty  of  verifying  the  presence  of  missiles  in  vertical  silos.  The  added  view- 
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ports  in  the  top  of  the  shelters  for  both  options  provided  easy  verification  of  those 
missiles  among  the  shelters.  Because  of  costs  and  technical  issues,  the  horizontal- 
shelter  option  was  selected  and  work  continued  on  refining  both  concepts  and 
cost  estimates. 

In  the  spring  of  1980  the  concept  of  elliptical  paths,  each  having  23  shelters, 
was  deleted  because  of  excessive  construction  costs,  and  a  grid  network  was 
substituted.  In  addition,  the  spacing  between  the  shelters  was  reduced.  The  net  ef¬ 
fect  of  these  changes  was  an  increase  in  procurement  funds  and  a  decrease  in  con¬ 
struction  funds.  The  design  and  cost  estimating  of  this  option  are  still  ongoing. 

In  addition  to  these  major  exercises,  the  program  office  completed  approx¬ 
imately  60  life-cycle  cost  estimates  between  July  1979  and  December  1980.  To  ac¬ 
complish  these  studies  within  limited  time  frames,  a  complete  and  flexible  com¬ 
puterized  cost  model  was  used. 

MX  Optimization  Cost  Model 

The  number  of  missiles  required  for  the  MX  system  is  established,  based  on 
the  threat  applicable  in  the  year  of  the  system’s  initial  operating  capability  (IOC). 
Thus,  the  threat  is  one  of  the  key  inputs  to  the  model  and,  when  coupled  with 
assumed  reliability  of  the  system,  provides  the  number  of  missiles  available  for 
launch  after  the  attack.  The  amount  of  surviving  megatonnage  was  established 
prior  to  design  of  the  system. 

The  program  designs  the  system  with  trade-offs  for  spacing,  hardness,  and 
other  variables,  and  produces  a  least-cost  solution  to  meet  the  specified  threat.  To 
do  this  the  computer  program  must  contain  information  on  the  unit  cost  of 
various  items. 

Research  and  development  costs  (3600)  are  through-put  to  the  model  because 
no  variations  owing  to  changes  in  the  number  of  units  were  known  or  an¬ 
ticipated.  These  factors  are  the  same  as  those  used  by  the  model  for  optimization; 
therefore,  it  is  not  necessary  to  have  the  model  estimate  them. 

Investment  costs  are  key  to  the  optimization  and  formed  the  basis  for  the  en¬ 
tire  model.  Non  -recurring  costs  are  estimated  off-line  and  through-put.  Recurring 
costs  are  estimated  by  the  model. 

The  first  unit  cost  and  learning-curve  slope  for  each  procurement  item  is 
calculated  off-line  using  cost-estimating  relationships,  engineering  buildup,  or  a 
combination  of  both.  Construction  costs  for  more  than  one  type  of  shelter  are 
estimated,  utilizing  subroutines  that,  in  effect,  design  the  shelter.  The 
subroutines,  given  the  spacing  requirements,  determine  the  hardness  of  the 
shelters  and  the  miles  of  road  required.  Requirements  for  various  materials, 
labor,  and  earth  moving  are  calculated  and  converted  to  costs  using  appropriate 
factors. 
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After  the  number  of  missiles,  required  manpower,  number  of  shelters,  and 
locations  have  been  determined,  the  operating  and  support  costs  are  calculated, 
and  the  computer  program  minimizes  costs  through  trade-offs  between  numbers 
of  misssiles,  shelter  hardness,  and  spacing  between  shelters. 

To  minimize  cost,  the  gradient  projection  method  devised  by  J.  B.  Rosen  is 
used  in  the  model.'  Rosen's  technique  modifies  the  method  of  steepest  descent  to 
permit  linear  constraints,  while  allowing  a  non-linear  objective  function.  The 
method  consists  of  first  finding  a  feasible  point,  calculating  the  gradient  of  the 
cost  function  at  that  point,  and  then  taking  a  step  along  the  steepest  gradient.  The 
length  of  the  step  is  the  maximum  permitted  by  the  physical  structure  of  the  cost 
surface  and  constraints.  If  the  step  falls  outside  the  feasible  region,  modifications 
are  performed  to  return  it  to  the  feasible  region.  Using  this  method,  convergences 
to  the  minimum  are  usually  quite  rapid;  however,  some  pathological  surfaces  will 
fail  to  converge  or  converge  very  slowly  unless  the  starting  point  is  chosen  with 
care. 

Through  use  of  the  model,  requirements  for  the  MX  missile  system  have  been 
derived  in  a  manner  unique  to  weapon  systems  acquisition.  The  requirements  are 
determined  by  a  specified  threat,  the  threat  is  quantified,  and  the  number  of 
missiles  required  to  counter  that  specific  threat  is  provided.  The  supporting  in¬ 
frastructure  for  the  weapon  system  has  been  constructed  to  require  the  least  life- 
cycle  cost.  Thus,  through  computer  modeling,  we  have  been  able  to  provide  time¬ 
ly  and  defensible  cost  estimates  on  many  proposed  MX  basing  and  launching 
modes.  H 
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Redistributing  Fixed 
Overhead  Costs 


Commander  Steve  ].  Balut,  USN 


^^hen  reviewing  Five  Year  Defense  Plan  (FYDP)  updates  submitted  by 
the  services,  we  in  the  Office  of  the  Secretary  of  Defense  (Program  Analysis  and 
Evaluation)  consider  alternatives  to  proposed  aircraft  procurement  quantities  and 
rates.  Costing  the  alternatives  involves  deriving  new  average  unit  prices  for  each 
lot  consistent  with  new  lot  quantities,  and  then  adjusting  prices  to  reflect  the  at¬ 
tendant  redistribution  of  fixed  overhead  resulting  from  the  change  in  production 
rate.  This  article  focuses  on  the  second  part  of  the  calculation,  which  we  refer  to 
as  a  "rate  adjustment." 

The  reason  for  shifts  in  overhead  are  not  well  understood,  and  predicting 
these  shifts  has  involved  more  art  than  science.  This  office  takes  a  first  cut  at  rate 
adjustments  using  a  heuristic  model  that  has  evolved  over  the  years. 

This  paper  evaluates  that  model  by  comparing  its  predictions  to  actual  con¬ 
tractor  experience,  and  then  presents  an  improved  model  derived  using  contrac¬ 
tor  data.  Use  of  the  new  model  is  illustrated  with  several  examples.  Finally,  a 
number  of  possible  extensions  to  this  research  are  suggested. 

Background 

During  FYDP  development  and  review,  we  are  frequently  asked  to  estimate 
the  cost  of  alternatives  to  aircraft  procurement  program  quantities  and  produc¬ 
tion  rates  proposed  by  the  services.  We  usually  base  our  estimates  on  detailed 
cost  sheets  submitted  by  the  services,  along  with  their  FYDP  updates.  These 
sheets  display  the  prices  the  services  say  they  will  pay  if  the  program  is  approved. 

We  in  OSD(PA&E)  generally  develop  our  estimates  in  two  steps: 
—Adjusting  the  service  estimate  for  quantity  change; 

—Adjusting  for  rate  change. 

The  first  step  is  straightforward  and  involves  developing  a  program  price  curve, 
at  the  recurring  flyaway  level,  and  repricing  the  new  lot  quantities,  accounting 
for  movements  up  or  down  the  learning  curve. 

The  second  step  in  making  the  estimate  is  more  difficult.  The  adjustment  for 
rate  change  involves  a  redistribution  of  indirect  charges  at  the  plant,  a  portion  of 
which  varies  with  activity  rate,  and  a  portion  of  which  doesn't.  Examples  of 
"variable  overhead"  costs  include  employee  benefits,  payroll  taxes,  and  other 
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production-related  indirect  costs  that  are  tied  to  the  number  of  direct  laborers 
working  in  the  plant  and  the  number  of  units  being  produced.  On  the  other  hand, 
costs  that  do  not  vary  with  activity  rate  and  are  fixed  in  the  short  term  are  usually 
referred  to  as  "fixed  overhead."  Examples  include  depreciation  and  amortization, 
insurance,  rent,  and  security. 

Unfortunately  for  the  cost  analyst,  there  are  some  indirect  expenses  that  don't 
fall  neatly  into  either  of  the  above  categories,  since  they  are  partially  fixed  and 
partially  variable.  We  refer  to  these  as  "semi-fixed  overhead."  An  example  is  utili¬ 
ty  expenses.  Lights  are  turned  on  and  environmental  conditions  maintained  at 
some  minimum  level  independent  of  the  activity  rate  (above  zero)  in  the  plant; 
however,  use  of  such  things  as  electricity,  heat,  and  air  conditioning  increases 
with  production  rate.  Utility  costs  (and  other  such  semi-variable  overhead  ex¬ 
penses)  are  gathered  and  reported  as  pools  and,  because  of  this,  the  fixed  and 
variable  portions  are  not  discernible. 

The  open  literature  contains  very  few  references  to  overhead  cost-estimating 
methods.  Gross  and  Diennemann  used  statistical  methods  to  show,  for  the 
aerospace  industry,  that  total  overhead  costs  were  62  percent  variable  with 
respect  to  direct  labor  dollars  and  46  percent  variable  with  respect  to  direct  labor 
plus  material.’  Hurta  shows  standard  labor  dollars  as  the  best  volume  measure¬ 
ment  for  a  cost  center,  and  presents  two  relationships  between  overhead  costs 
and  standard  labor  dollars  for  use  in  estimating  overhead  cost.^ 

Other  documents  provide  definitions  of  overhead  cost  categories,  provide 
guidance  for  monitoring  such  costs,  and  discuss  the  relationship  between  cost  ac¬ 
counting  and  price. 5  Two  references  present  interesting  overhead  cost  manage¬ 
ment  techniques.  The  first  proposes  a  sharing  arrangement  for  overhead  cost 
underruns  and  overruns  of  advanced  agreement  targets. <  The  other  presents  a 
method  for  allocating  overhead  charges  on  the  basis  of  a  mathematical  program- 
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ming  model  for  production  and  sales  possibilities.^  The  basic  scheme  is  to  charge 
products  on  the  basis  of  their  use  of  scarce  resources  of  the  firm  (charges  being  the 
value  of  the  dual  variables  of  the  profit-maximizing  mathematical  program). 

When  making  our  estimates,  we  treat  overhead  statistically,  not  functionally. 
In  this  way,  we  do  not  get  bogged  down  with  allocation  schemes  for  splitting 
semi-variable  overhead  pools.  Instead,  we  observe  changes  in  total  overhead  ex¬ 
penditures  associated  with  changes  in  activity  rate  within  the  plant,  and  use  these 
observations  as  guides  for  the  future.  Application  of  this  experience  has  evolved 
over  the  years  into  the  form  of  a  mathematical  model  that  we  refer  to  as  a  "rate 
adjustment"  model. 

It  is  important  to  note  again  that  we  develop  our  estimates  in  two  steps.  The 
first  involves  the  standard  use  of  learnng  curve  theory,*’  the  application  of  which, 
at  the  price  level,  involves  an  implicit  assumption  that  overhead  is  100  percent 
variable  with  direct  costs.  Our  second  step  corrects  for  this  erroneous  assumption 
as  application  of  the  rate  adjustment  model  (to  the  estimate  derived  in  step  one) 
accounts  for  the  redistribution  of  fixed  overhead  across  the  new  activity  levels 
within  the  plant.  (See  following  page.) 
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The  model  is: 


Old 

Qi 


New 

Qi 


pR  +  a-  PR) 


Where  i  =  lot  number: 


Fi 


New 

Qt 


p 

R 


=  The  factor  used  to  adjust  the  estimate  for  lot  i  derived  in  step 
one. 

=  The  quantity  of  aircraft  in  lot  i  in  the  basic  service  program. 

=  The  new  quantity  for  lot  i  in  the  alternative  program. 

=  The  fraction  of  price  represented  by  overhead. 

=  The  fraction  of  overhead  that  is  fixed  in  the  short  term. 


The  product  PR  represents  the  fraction  of  price  represented  by  the  overhead 
fixed  in  the  short  term.  Lot  quantity  is  used  as  a  proxy  for  lot  direct  cost.  When 
the  new  quantity  (associated  with  the  alternative  program)  is  greater  than  the  old 
quantity,  the  ratio  QPld/QNew  is  less  than  one,  resulting  in  a  factor  Fj  less  than 
one.  When  applied  to  the  estimate  developed  in  step  1,  the  factor  reduces  unit 
cost  by  spreading  fixed  overhead  over  a  larger  quantity.  When  the  new  quantity 
is  less  than  the  old,  the  opposite  occurs. 

The  values  used  for  P  and  R,  .35  and  .42,  are  estimated  aerospace  industry¬ 
wide  averages  derived  by  this  office  about  2  years  ago. 

Figure  1  shows  a  graph  of  the  factor  Fi  as  a  function  of  the  ratio  Q?^ew/Qpld. 
This  unit  of  measure  is  chosen  for  the  x-axis  because  it  allows  direct  reading  of  the 
percentage  increase  or  decrease  of  the  new  quantity  with  respect  to  the  old. 

Our  experience  indicates  that  rate  adjustment  made  using  this  model  are  good 
when  the  ratio  QNew/QOld  is  near  unity;  however,  odr  confidence  decreases 
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new  old 

Q  /  Q 


rapidly  as  the  ratio  deviates  from  unity.  We  have  applied  the  model  to  make  first- 
cut  estimates,  cautiously,  when  the  ratio  remained  within  the  limits: 


.5< 


qNcw 

QOld 


2.0 


When  quantity  changes  varied  more  drastically  than  that,  other  methods  v?ere 
used. 

Reservations  about  the  accuracy  of  this  model  led  to  the  analysis  reported  in 
this  article. 

Contractor  data  were  collected  for  two  purposes: 

—To  compare  model  predictions  with  actual  contractor  experience,  and 
—To  use  the  data  to  develop  an  improved  model. 
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The  next  several  sections  report  the  results  of  the  data-gathoring  effort,  old  model 
evaluation,  and  new  model  derivation,  followed  by  several  examples  that  il¬ 
lustrate  its  use.  Then  the  limits  of  applicability  of  the  model  are  discussed,  fol¬ 
lowed  by  the  presentation  of  a  number  of  promising  extensions  to  this  research  ef¬ 
fort 

Data 

Data  was  extracted  from  contractor  cost  data  report  (CCDR)  plant-wide  data 
reports  (DD  Form  1921-3).  To  date,  only  three  contractors  all  manufacturers  of 
fighter/attack  aircraft,  have  submitted  reports  containing  data  suitable  for 
analysis.  The  1921-3  report  presents  annual  overhead  charges  associated  with 
contractor-reported  annual  business  bases.  (The  reported  business  bases  general¬ 
ly  correspond  to  actual  business  bases;  however,  there  are  indications  that  they 
may  differ  by  1  or  2  percent  in  some  instances,  due  to  lack  of  uniformity  among 
contractors  on  how  the  base  is  described.)  The  reports  used  in  this  analysis  pre¬ 
sent  business  base  and  overhead  for  years  1975,  1976,  and  1977,  and  projections 
for  1978,  and  in  one  case,  beyond  1978. 

For  purposes  of  this  analysis,  cost  elements  were  aggregated  into  the  following 
categories: 

—Direct  labor 
—Direct  material 
—Indirect  labor 


TABLE  1 

Cost  Breakdown  (Percentages) 


Contractor 

1 

2 

3 

ALL 

Direct 

■■i 

Labor 

20 

17 

24 

Material 

41 

40 

50 

WSM 

Total 

61 

57 

74 

mm 

indirect 

■■ 

Labor 

25 

30 

Material 

3 

3 

GAA 

11 

10 

8 

10 

Total 

39 

43 

26 

38 

■'I 
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—Indirect  material 

—General  and  administrative  (G&A) 

In  this  analysis,  the  term  "cost"  refers  to  the  sum  of  direct  and  indirect  costs  plus 
G&A  expense.  The  term  "price"  refers  to  cost  plus  fee,  where  fee  is  estimated  to 
be  11  percent  of  the  sum  of  all  categories  shown  above,  less  direct  material.  When 
making  cross-year  comparisons,  all  data  were  reduced  to  1975  dollars  using 
OSD(C)  procurement  escalation  indices. 

Table  I  shows  a  summary  cost  of  breakout  by  contractor.  The  numbers  in  the 
table  represent  percentages  of  total  cost. 

Data  in  this  sample  indicate  that,  for  the  prime  contractor: 

— Overhead  represents  about  38  percent  of  cost. 

—Labor  overhead  is  about  132  percent  of  direct  labor  cost. 

—Material  overhead  is  about  7  percent  of  direct  material  cost. 

This  analysis  indicates  that  cost  represents  about  94  percent  of  price,  the  re¬ 
mainder  being  fee. 


FIGURE  2 


Old  Rate  Adjustment  Model  Compared  to  Actual  Data 


new  old 


Q  /Q 


70  I  Concepts 


Old  Model  Compared  to  Data 

Figure  2  presents  a  graph  of  the  data,  along  with  the  graph  of  the  model. 
Eleven  data  points  were  obtained,  one  very  close  to  the  point  where  QNew/QOld 
equals  one,  five  to  the  left  of  that,  and  five  to  the  right.  The  range  of  the  sample  is 


.83  < 


Q  New 


<  1.85 


representing  from  a  1 7  percent  reduction  up  to  an  85  percent  expansion  in  direct 
business.  The  plot  indicates  that  the  model  has  approximately  the  right  cur¬ 
vilinear  form,  but  the  wrong  slope.  According  to  this  sample,  our  model  tends  to 
underestimate  the  effect  of  reductions  in  activity  rate  on  per-unit  overhead  costs, 
and  overestimates  the  effect  of  expansions. 

Improved  Rate  Adjustment  Model 

It  was  clear  from  the  start  that  the  model  could  be  improved  by  adding  an  ex¬ 
ponent  to  the  ratio  term  as  follows; 


ff  = 


„  New 

^  I 


b 

PR+  {l-  PR) 


where  all  variables  are  as  defined  earlier,  and  exponent  b  has  been  added.  We 
have,  in  simplified  form: 


1/  .  =  a  +  (1  -  a) 

I  I 

A  non-linear  curve-fitting  routine  was  applied  with  the  following  results: 


Fitted 

standard 

Coefficient 

Value 

Error 

T-Statistic 

a 

0.162 

0.043 

3.76 

b 

-1.669 

0.54 

-3.1 
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This  form  produced  very  favorable  statistics:  an  adjusted  coefficient  of  deter¬ 
mination  of  0.95,  a  standard  error  of  estimate  (SEE)  of  0.012,  and  a  Durbin 
Watson  statistic  (D-W)  of  2.46.  Table  II  compares  actual  and  predicted  values  for 
the  dependent  variable. 

The  predicted  values  differ  from  the  actual  values  at  worst  by  less  than  2  percent, 
and  on  the  average  are  about  1  percent  off.  Figure  3  shows  a  graph  of  the  model, 
along  with  the  plotted  data,  and  also  a  trace  of  the  model  shown  in  Figure  1. 

Coefficient  "a"  stands  for  the  product  PR,  the  fraction  of  price  represented  by 
overhead  that  is  fixed  in  the  short  term.  Analysis  of  contractor  data  revealed  that 
P,  the  fraction  of  price  represented  by  overhead,  is  about  0.36.  The  data  also  im¬ 
ply,  via  the  least-squares,  curve-fitting  routine,  that  coefficient  a,  and  therefore, 
product  PR,  equals  0.162.  This  implies  that  R,  the  fraction  of  overhead  that  is 
fixed  in  the  short  term,  equals  about  0.45.  These  results  compare  favorably  with 
the  findings  of  an  undocumented  study  done  by  this  office  2  years  ago  in  which 
aerospace  industry-wide  averages  for  P and  R  were  estimated  to  be  0.35  and  0.42, 
respectively. 

I  ' 


TABU  II 

Comparison  of  Actual  Data  With  Model  Predictions 
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FIGURE  3 

New  Rate  Adjustment  Model 
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Examples 

The  use  of  the  rate  adjustment  model  is  illustrated  here  with  two  simple,  con¬ 
trived  examples.  The  first  illustrates  the  technique  when  the  contractor  has  only 
one  program.  The  second  example  shows  the  modification  necessary  when  the 
contractor  has  other  ongoing  programs. 

Data  for  example  1  are  shown  in  Table  111.  Line  one  indicates  that  the  basic  pro¬ 
gram  has  a  level  production  rate  of  10  per  period,  and  the  alternative  program, 
indicated  by  line  two,  specifies  a  proposed  reduction  in  the  first  lot  quantity  from 
10  to  8,  and  an  increase  in  the  third  lot  quantity  from  10  to  15.  The  estimates  of 
unit  recurring  flyaway  prices  for  the  new  lot  quantities,  assuming  overhead  is  100 
percent  variable  with  direct  costs,  are  shown  in  line  three.  (Derivation  of  these 
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TABLE  III 

Data  for  Example  1 


LOTS 

Line 

Description 

1 

2 

3 

B 

Old  program  Q 

10 

10 

10 

B 

New  program  Q 

S 

10 

IS 

1 

Estimated  avg  unit  recurring 
flyaway  price  (without  rate 
adjustment) 

11 

10 

9 

B 

Rate  adjustment  factor 

1.073 

1.0 

.920 

5 

Estimated  avg  unit  recurring 
flyaway  price  (after  rate 
adjustment) 

11.8 

10 

8.3 

6 

Adjusted  recurring  flyaway 
price 

94.4 

100 

_ 

124.2 

estimates  is  not  addressed  in  this  paper.)  Line  four  shows  the  rate  adjustment  fac¬ 
tors  derived  using  the  model 

New] 

^  i 

f  ,•  =  .162  - +  .838 

‘  Q  Old 


Line  five  is  the  product  of  lines  three  and  four,  and  represents  the  average  unit 
recurring  flyaway  price  after  adjusting  for  change  in  rate.  The  last  line  is  the 
product  of  lines  five  and  two,  and  represents  the  recurring  flyaway  prices  of  each 
lot. 

For  lot  1,  a  20  percent  reduction  in  production  rate  leads  to  about  a  7  percent 
increase  in  unit  price  due  to  redistribution  of  fixed  overhead.  For  lot  three,  a  50 
percent  increase  in  rate  leads  to  an  8  percent  decrease  in  unit  price. 

The  data  for  example  2  are  shown  in  Table  IV.  In  this  case,  we  assume  the  con¬ 
tractor  has  other  ongoing  programs  which  contribute  to  his  business  base.  The 
other  programs  must  be  taken  into  account  when  redistributing  fixed  overhead. 
The  procedure  is  similar  to  that  used  in  example  1,  with  the  exception  of  the 


74 


Concepts 


II 


TABLE  IV 

Data  for  Example  2 


LOTS 

Line 

Description 

1 

2 

3 

1 

Old  program  Q 

10 

10 

10 

2 

Equivalent  units  of  other 
business 

/ 

5 

10 

IS 

3 

Old  rate  (in  equivalent 
units) 

IS 

20 

25 

4 

New  program  Q 

8 

10 

15 

5 

New  rate  (in  equivalent 
units) 

13 

20 

30 

6 

Estimated  avg  unit  recurring 
flyaway  price  (without  rate 
adjustment) 

11 

10 

9 

7 

Rate  adjustment  factor 

1.044 

1.0 

.957 

8 

Estimated  avg  unit  recurring 
flyaway  price  (after  rate 
adjustment) 

11.5 

10 

8.6 

■ 

Adjusted  recurring  flyaway 
price 

91.9 

100 

129.3 

derivation  of  the  ratio,  QNew/Q01d_  used  in  the  model.  In  example  1,  we  al¬ 
lowed  program  quantities  QNew  and  to  represent  old  and  new  production 
rate.  In  example  2,  we  translate  the  contractor's  other  direct  business  into 
equivalent  units^  of  the  article  being  priced,  and  add  these  annual  subtotals  to 
both  QNew  and  QOI'^.  Now  the  ratio  captures  the  change  in  total  contractor  ac¬ 
tivity  level  resulting  from  a  change  in  our  program: 

QNew  -I-  Equivalent  units  of  other  business 

QOId  -1-  Equivalent  units  of  other  business 


7.  Divide  other  direct  business  by  the  unit  price  of  the  product  being  considered. 
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The  data  for  example  2  in  Table  IV  are  essentially  the  same  as  for  example  1, 
but  a  few  more  lines  have  been  added.  Line  two  shows  the  number  of  equivalent 
units  of  other  business.  This  is  added  to  lines  one  and  four  to  arrive  at  old  and 
new  rates,  in  equivalent  units,  as  shown  in  lines  three  and  five.  The  ratio  of  these 
figures  is  entered  into  the  model  to  arrive  at  the  rate  adjustment  factors  shown  in 
line  seven. 

Note  that  redistribution  of  fixed  overhead  over  a  large  business  base  results  in 
a  smaller  adjustment  than  seen  in  example  1.  A  20  percent  decrease  in  our  pro¬ 
gram  leads  to  about  a  4  percent  increase  in  unit  price,  as  compared  to  about  7  per¬ 
cent  in  example  1.  For  lot  three,  a  50  percent  increase  in  our  program  leads  to 
about  a  4  percent  decrease  in  unit  price,  as  compared  to  about  8  percent  in  example  1. 

The  data  used  to  develop  the  model  presented  in  this  paper  relate  to 
fighter/attack  airframe  manufactures.  Applying  the  model  at  the  flyaway  price 
level  involves  an  implicit  assumption  that  the  shift  in  business  base  will  affect 
propulsion  and  avionics  contractor  costs  (that  can  add  up  to  about  48  percent  of 
flyaway)  in  a  manner  similar  to  the  effect  on  the  airframer's  costs.  In  cases  where 
the  engine  and/or  avionics  manufactures  have  much  larger  business  bases,  as 
compared  to  the  airframers,  the  effect  of  program  quantity  changes  could  be 
made  separately  for  each  major  contractor.  In  any  event,  the  model  should  be  ap¬ 
plied  with  caution,  and  then  only  within  the  range  of  data  supporting  its  develop¬ 
ment  (.83  <  QNew/Q01d<  i.gs). 

Extensions 

The  direct  and  immediate  extensions  to  this  research  are  plentiful.  1  will  men¬ 
tion  but  a  few. 

OVERHEAD  DATA 

The  plant-wide  data  report  (CCDR  Form  1921-3)  is  the  only  known  com¬ 
prehensive  source  of  overhead  expenditures  by  defense  contractors,  aside  from 
reports  received  by  the  Defense  Contract  Administration  Services  (DCAS),  some 
irregular  government  audit  reports,  and  actual  visits  to  the  contractors.  The  re¬ 
quirement  to  submit  the  1921-3  report  has  been  in  effect  only  a  few  years.  There 
are  no  reports  covering  the  period  before  1975,  and  little  more  than  a  handful 
covering  1975  to  the  present.  This  probem  will  be  overcome  as  more  and  more 
reports  are  received;  however,  there  exists  a  fundamental  problem  with  data  con¬ 
sistency  between  reporting  contractors.  Review  of  reports  at  hand  reveal  major 
inconsistencies,  including: 

—All  contractors  report  plant-wide  data,  with  the  exception  of  one,  who  submits 
separate  reports  on  each  program  at  the  plant. 

—Some  contractors  include  material  costs  in  the  overhead  base  and  others  ex¬ 
clude  them. 
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— Some  contractors  use  one  set  of  cost  categories  when  reporting  historical  costs, 
and  another  in  their  projections. 

The  1921-3  report  format  and  reporting  procedures  need  to  be  firmed  up  and  en¬ 
forced,  or  the  value  of  the  data  received  on  these  reports  will  be  seriously  degraded. 

EXTEND  THE  MODEL  DATA  BASE 

Only  11  data  points  were  used  to  develop  the  model  reported  here.  Con¬ 
fidence  in  the  model  could  be  improved,  and  its  range  and  applicability  could  be 
extended,  if  more  data  points  were  obtained.  As  additional  data  is  developed, 
separate  models  could  be  derived  for  cargo  aircraft,  helicopters,  missiles,  space 
vehicles,  and  perhaps  even  tanks  and  ships.  In  addition,  we  need  to  look  carefully 
at  the  characteristics  of  engine  manufacturers  and  avionics  manufacturers.  When 
sufficient  data  are  available,  separate  models  can  be  derived  from  each  major 
defense  contractor. 

DEVELOPMENT  PROGRAM  OVERHEAD  COSTS 

Overhead  costs  associated  with  development  programs  are  less  well 
understood  than  those  for  production  programs.  Some  original  thought  should 
be  given  this  area.  It  seems,  at  the  very  least,  that  a  review  of  recent  development 
program  costs  would  reveal  the  relationship  between  direct  and  indirect  costs 
during  development. 

REPRICING  MODEL 

If  our  rate  adjustment  model  were  coupled  with  a  learning  curve  routine,  the 
combination  could  perform  the  two  steps  in  repricing  a  new  program  quantity 
stream  in  an  efficient  and  consistent  manner.  This  "repricing  model"  is  simple 
enough  to  be  programmed  on  a  hand  calculator.  || 


Three  views  of  the  Impact  of  Production  Rate  Changes— II 

Effect  of  Production 
Rate  on  Weapon 
System  Cost 

Dr.  Charles  H.  Smith 

l^ecent  experience  has  shown  that  production  rates  for  new  military 
weapon  systems  are  subject  to  continual  adjustment  during  both  the  planning  and 
production  phases.  Yet  the  impact  of  these  rate  changes  on  unit  costs  is  not 
generally  understood.  The  learning-curve  concept,  which  estimates  unit  cost  as  a 
decreasing  function  of  cumulative  production  quantity,  is  heavily  used  in  cost 
estimating.  Seldom,  however,  are  adjustments  made  to  reflect  the  impact  of  dif¬ 
ferent  production  rates.  This  paper  addresses  the  current  state  of  production  rate 
research  and  proposes  a  technique  applicable  to  future  weapon  system  cost 
analysis. 

We  must  remember  that  factors  other  than  cost  must  be  given  weight  in  pro¬ 
duction  rate  decisions.  For  example,  rate  decisions  affect  deployment  and 
readiness;  having  a  tank  available  this  year  is  not  the  same  as  having  it  available 
next  year.  Training  and  logistics  factors  also  must  be  considered.  Unlike  most 
commercial  production,  the  typical  weapon  system  is  designed  for  a  build-up  to 
high  quantity  production  for  a  relatively  few  years.  Typically,  follow-on  produc¬ 
tion  will  then  continue,  but  at  much  lower  rates,  which  is  one  reason  that  excess 
capacity  often  exists. 

Affordability  issues  also  prevent  the  viewing  of  production  rate  as  an  isolated 
cost  optimization  problem.  Affordability  issues  arise  from  an  overall  constraint 
on  annual  spending.  This  constraint,  combined  with  large  numbers  of  competing 
programs,  causes  production  stretch-outs  and  other  adjustments.  Military 
preparedness  may  thus  dictate  that  almost  no  system  can  be  produced  at  rates 
that  would  be  most  efficient.  Factors  leading  to  development  of  a  production  plan 
are  sketched  in  Figure  1. 

Although  the  reasons  for  a  particular  production  plan  may  seem  convincing, 
a  complete  cost  analysis  requires  an  estimate  of  the  unit  cost  effect  of  alternative 
production  rate  schedules.  While  an  unlimited  variety  of  program  circumstances 
exist,  the  rate  problems  fall  into  two  basic  classes.  The  division  is  based  on 
whether  or  not  the  system  under  analysis  is  past  the  preproduction  planning 
stage.  The  two  situations  present  different  factors  with  which  concerned, 
for  flexibility  is  hampered  once  manufacturing  facilities  are  on  ■  '  md  workers 

are  hired.  Moreover,  the  first  problem — the  planning  pr  ’  ni-  marily  con¬ 
cerned  with  the  cost  over  the  entire  production  life  ( '  piograni.  he  second 
problem  is  more  oriented  toward  assessing  the  cost  and  pioduction  impact  of 
short-term  budget  adjustments. 
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FIGURE  1 

Factors  Leading  to  Development  of  a  Production  Plan 


Several  research  studies  have  considered  the  effect  of  rate  on  weapon-system 
cost.  An  extensive  review  of  these  studies  was  conducted  as  an  initial  step  in  a  re¬ 
cent  study  conducted  by  the  U.S.  Army  Procurement  Research  Office.’  Only  a 
few  key  studies  and  their  findings  will  be  mentioned  here. 

A  1974  RAND  study  attempted  to  modify  the  usual  learning-curve  approach 
by  testing  models  across  airframe  programs  with  an  additional  rate  term.^  These 
researchers  were  forced  to  conclude  that  a  general  cost-rate  relationship  could  not 
be  predicted  with  any  degree  of  confidence.  For  advanced  planning  purposes  they 
recommended  ignoring  rate  effects  in  aircraft  production  programs  because  they 
were  dominated  by  other  uncertainties.  Note  that  the  researchers  did  not  claim 
that  rate  does  not  have  an  important  effect,  only  that  the  rate  effect  was  not 
predictable  with  the  models  tested.  Although  confined  to  airframes,  this  work 
suggests  that  it  is  unrealistic  to  expect  to  obtain  a  single  general  rate-cost  model, 
even  within  a  product  class. 

Some  Air  Force  researchers  have  had  modest  success  with  a  different  ap¬ 
proach.’  Rather  than  attempt  to  determine  a  general  model,  these  researchers  fit  a 
regression  equation  with  a  production  rate  term  to  the  historical  data  for  a  single 
program.  They  typically  obtained  reasonably  good  fits  and  a  statistically  signifi¬ 
cant  rate  term.  Although  they  applied  the  model  to  direct  labor  costs,  the  tech¬ 
nique  could  be  used  for  total  production  costs  and  applied  in  any  product  area. 


1.  C.  H.  Smith,  Production  Rale  and  Weapon  System  Cost:  Research  Review,  Case  Studies,  and 
Planning  Model,  APRO  80-05,  Fort  Lee,  Va.:  U.S.  Army  Procurement  Research  Office,  1980. 

2.  J.  P.  Large,  et  al..  Production  Rate  and  Production  Cost.  R-1609-PAjiE,  Santa  Monica,  Calif.: 
RAND  Corporation,  1974, 

3.  See,  for  example,  the  following:  L.  L.  Smith,  An  Investigation  of  Changes  in  Direct  Labor  Re¬ 
quirements  Resulting  from  Changes  in  Airframe  Production  Rate.  Unpublished  Ph.D.  Dissertation, 
University  of  Oregon,  1976. 
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FIGURE  2 

A  Few  Possible  Behaviors  for  Overhead  Cost  Elements 


The  principal  drawback  to  this  approach  is  the  extensive  data  base  required  on  a 
specific  program  before  the  model  can  be  used.  Most  key  decisions  are  made 
before  such  a  data  base  can  exist.  The  model  has  no  generality  because  wide 
variations  in  parameter  values  have  occurred  for  systems  with  similar  production 
quantities  and  rates. 

The  general  conclusions  that  can  be  drawn  from  the  existing  empirical 
research  base  are  limited.  It  is  clear  that  rate  affects  unit  costs  and  that  the  specific 
effect  is  highly  dependent  on  program  specific  features.  Typically,  unit  cost 
estimates  are  much  less  sensitive  to  rate  uncertainty  than  to  total  program  quanti¬ 
ty  uncertainty.  Finally,  for  the  planning  problem  there  is  evidence  that  the 
overhead  cost  element  frequently  accounts  for  a  much  greater  proportion  of  the 
rate  effect  than  do  inefficiencies  in  the  use  of  material  or  labor.  A  few  possible 
behaviors  of  overhead  cost  elements  as  a  function  of  rate  are  shown  in  Figure  2. 

The  conduct  of  empirical  research  in  this  area  is  fraught  with  difficulties.  In 
dealing  with  historical  rate  data  it  may  be  impossible  to  know  the  rate  for  which 
the  facility  was  designed.  It  also  may  not  be  possible  to  know  the  amount  of  time 
available  to  the  contractor  for  planning  and  adapting  to  a  rate  change.  Likewise, 
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the  optimal,  as  distinct  from  the  planned,  rate  may  not  be  known.  Some  ap¬ 
proaches  to  data  gathering  invite  bias.  This  is  especially  true  of  techniques  using 
contractor-supplied  data  to  which  the  contractor  has  not  been  contractually  com¬ 
mitted.  Thus,  contractor  historic  data  is  preferred  because  it  is  an  objective 
measure  of  what  actually  happened.  In  the  cost-rate  area  one  also  risks  searching 
for  fine  differences  concealed  by  larger  system  disturbances. 

In  our  own  research,  several  missile  systems  were  investigated  to  gain  addi¬ 
tional  insight  into  the  effects  of  production  rate  on  system  cost.  The  six  missile 
systems  examined  covered  a  broad  spectrum  of  conditions.  Included  were  cases 
with  very  long  production  histories,  and  cases  only  now  in  production  planning 
for  low-rate  initial  production.  The  cases  also  varied  from  low-volume,  labor- 
intensive  production  such  as  Pershing  to  high  volume,  highly  automated  produc¬ 
tion  such  as  that  planned  for  the  multiple  launch  rocket  system  (MLRS).  One  case 
had  steady  rates  during  production  periods  but  was  plagued  with  breaks  in  pro¬ 
duction.  Other  cases  had  U.S.  Army  rate  effects  highly  buffered  by  foreign 
military  sales.  One  system  was  experiencing  an  extended  period  of  low-rate  pro¬ 
duction  because  an  improved  version  of  the  system  was  on  the  way.  Such  a  varie¬ 
ty  of  circumstances  found  within  even  a  single  product  class  makes  generalization 
difficult. 

Nevertheless,  these  investigations  led  us  to  recommend  certain  actions.  For 
example,  in  agreement  with  earlier  work,  it  was  suggested  that  low-range  plan¬ 
ners  focus  on  military  requirements  without  regard  to  rate  behavior.  At  that 
stage,  rates  and  their  effects  are  unpredictable  and  dominated  in  importance  by 
more  fundamental  choices.  As  appropriate,  the  government  should  require  con¬ 
tractors  to  explain  as  part  of  their  proposals  the  mechanisms  for  accommodating 
rate  changes.  This  action  should  encourage  manufacturing  flexibility  in  adapting 
to  rate  changes.  Since  the  size  and  structure  of  the  defense  industry  are  ultimately 
major  factors  in  production  costs,  further  study  should  be  conducted  of  the  op¬ 
timal  structure  of  the  defense  industry  arid  its  relationship  to  the  defense  budget. 
Finally,  a  simple  model  based  on  overhead  considerations  should  be  used  for  pro¬ 
gramming  and  budgeting  estimates  under  rate  changes. 

Several  theoretical  cost-rate  models  have  been  proposed.  While  the  models 
differ  in  several  respects,  most  either  address  the  wrong  problem  or  require  the 
estimation  of  too  many  difficult,  detailed  parameters  for  top-level  planners.  None 
has  seemed  suitable  for  use  as  a  general  tool  for  budget  planners.  One  proposed 
estimation  model  is  suggested  below.  It  seems  to  provide  a  degree  of  insight  for 
many  cases  with  limited  data  requirements.  The  model  is  based  on  the  premise 
that  costs  dependent  on  time  rather  than  cumulative  quantity  are  the  major  ex¬ 
plainers  of  the  cost  impact  of  rate  changes. 

To  illustrate  the  approach,  consider  program  planning  for  a  system  prior  to 
the  beginning  of  actual  production.  Suppose  that  a  total  quantity  of  Q  units  of 
the  system  will  be  bought.  What  is  the  estimated  cost  per  unit  if  the  system  is  pro¬ 
cured  uniformly  over  T  years? 
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Let  the  variable  costs  y  for  the  x  unit  be  given  by  y  ^  ax  ^  where  a,  b>0.  If 
the  acquisition  period  is  T  years,  then  the  annual  production  rate  is  Q/T.  Let  the 
annual  fixed  costs  allocable  to  the  progam  be  constant  and  denoted  by  F.  Then 
the  average  unit  production  cost  A  for  all  production  is  given  by 

Q 

E  ak-^+T  F  (^) 
k=l  ^  1-b 

Q  Q 

This  model  is  useful  for  estimating  the  average  unit  cost  differences  for  pro¬ 
curing  the  same  total  quantity  under  different  rates.  It  is  easy  to  modify  the  model 
to  accommodate  fixed  costs  that  change  from  year  to  year,  provided  these  can  be 
estimated.  Likewise,  a  slight  modification  is  required  to  deal  with  those  cases 
where  substantial  production  experience  is  acquired  for  units  not  sold  to  the 
analyzing  buyer,  e.g.,  foreign  military  sales.  Discount  factors  can  also  be  incor¬ 
porated  as  needed. 

The  following  example  illustrates  the  use  of  the  model  and  the  predicted  effect 
on  costs. 

Example.  Two  production  plans  under  consideration  are  given  in  Table  1. 
Both  options  result  in  an  overall  production  volume  of  120,000  units.  Let  the 
variable  costs  V(k)  for  both  plans  be  expressed  by  the  following 

Variable  cost  for  unit  =  V(k}  =  10,000fc“-^^^. 

This  expression  assumes  a  90  percent  learning  curve  and  describes  variable  cost 
reductions  as  a  function  of  the  cumulative  quantity  produced.  Also,  let  the  an¬ 
nual  fixed  costs  of  maintaining  the  program  be  $10  million. 
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120,000 

(  I  10000/c'-^‘‘’2)  +  3(10^^ 

- $4493. 

120,000 


In  this  hypothetical  example  the  estimated  average  unit  cost  decrease  is  $2,500 
when  production  follows  Option  2  instead  of  Option  1.  Thus,  in  this  example, 
doubling  the  production  rate  for  the  entire  program  should  save  36  percent  of 
program  production  costs.  Of  course,  in  real  life,  decision-making  factors  such  as 
changes  in  military  value  dependent  on  time  of  production  must  be  considered. 

In  the  above  example  the  model  was  used  to  project  the  cost  effect  of  rate  op¬ 
tions  on  a  program  in  the  production  planning  stage.  Frequently,  however,  the 
principal  concern  is  to  estimate  the  cost  effect  of  a  rate  change  for  next  year  on  a 
current  production  effort.  A  variation  in  the  above  model  handles  this  problem. 

The  real  test  of  the  usefulness  of  the  model  is  the  accuracy  of  its  predictions. 
One  indicator  of  that  accuracy  is  described  below.  Only  one  study  was  found 
that  analyzed  rate  effects  in  sufficient  detail  to  permit  comparison.  A  detailed  ex¬ 
amination  was  made  of  the  effects  of  rate  changes  on  MLRS  costs."  The  research¬ 
ers  analyzed  in  detail  the  capital  equipment  changes  and  other  items  required  by 
rate  adjustments.  This  study  was  only  a  forecast  of  cost  effects  because  no  pro¬ 
duction  had  taken  place.  The  study  finding  must  not  be  taken  as  current  projec¬ 
tions  of  MLRS  behavior  because  the  contractor  analyzed  was  not  the  winner  of 
the  competitive  selection  process.  Nevertheless,  the  approach  taken  was  very 
detc’,  '  and  radically  different  from  the  simple  model  proposed  here.  It  stands  as 
one  i  of  the  predictive  ability  of  the  model.  The  accuracy  of  the  model  in 
matching  the  detailed  study  findings  is  a  matter  of  judgment.  Moreover,  one 
detailed  comparison  does  not  suffice  for  arriving  at  conclusions.  More  such  case 

4.  Analysis  of  the  Effects  of  Production  Rate  and  Quantity  Changes  on  Production  Costs,  Science 
Applications,  Inc.  (Prepared  for  MLRS  Project  Office  under  Contract  No,  DAAK40-79-D-0(XX)6), 
Huntsville,  Ala.,  1980. 
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analyses  are  required  to  increase  confidence  in  such  models.  In  this  case, 
however,  the  model  seemed  to  anticipate  the  effect  of  rate  changes. 

It  is  obvious  that  the  fixed-cost-based  model  is  not  totally  accurate.  It  ap¬ 
pears,  however,  that  use  of  such  a  model  would  refine  current  macro-level  budget 
practices.  The  model  should  not  be  strained  too  hard  by  employing  it  carelessly 
over  too  great  a  range  of  rates.  The  accuracy  of  these  simple  tools  probably  falls 
rapidly  after  much  more  than  a  SO  percent  rate  increase  or  100  percent  rate 
decrease.  In  the  short-term  analysis  one  must  recognize  a  greater  amount  of  fixed 
costs.  One  reason  is  the  unwillingness  of  a  firm  to  cut  back  labor  in  proportion  to 
the  short-term  rate  cut. 

Finally,  any  of  these  cost  models  provides  information  along  only  one  dimen¬ 
sion  of  the  overall  problem.  Changing  rate  means,  in  a  sense,  procuring  a  dif¬ 
ferent  system,  for  the  military  value  of  a  system  is  a  function  of  its  availability. 
Simple  arguments  based  on  willingness  to  incur  total  system  costs  could  often 
suggest  that  stretch-out  is  much  more  costly  in  terms  of  estimated  military  value 
than  in  terms  of  production  costs.  Too  much  relative  emphasis  is  typically  placed 
on  the  cost  side  of  the  cost-value  spectrum,  simply  because  it  is  more  easily  quan¬ 
tified. 

The  suggestions  below  are  unproven,  but  they  represent  reasonable  decision 
rules  today.  Decisions  are  grouped  here  into  the  three  classes  of  advanced  plan¬ 
ning,  budgeting,  and  negotiations.  It  is  proposed  that  rate  effects  be  ignored  for 
advanced  planning  purposes.  In  the  budgeting  phase  one  can  use  a  rough  rule  of 
thumb  that  suggests  a  per-unit  price  rise  of  0-10  percent  for  a  downward  rate  ad¬ 
justment  of  50  percent.  The  exact  percentage  used  should  be  biased  toward  one 
extreme  or  the  other  based  on  system  specific  information.  If  sufficiently 
reasonable  estimates  are  available,  the  model  suggested  above  could  be  applied  at 
this  stage.  More  data  may  be  available  in  the  contract  negotiation  stage.  If  suffi¬ 
cient  production  history  exists  for  the  system,  applying  a  program  specific  regres¬ 
sion  technique  is  recommended. 

A  fuller  understanding  of  the  contingent  nature  of  the  rate  problem  is  a 
massive  undertaking  whose  value  depends  on  the  cost-rate  effects  on  decision¬ 
making.  Readers  are  invited  to  contribute  to  the  advancement  of  this  research 
area  through  added  empirical  knowledge  of  additional  commodities  and  through 
conceptual  advances.  || 
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1  he  issue  of  production  rates  for  defense  systems  has  recently  been  given  in¬ 
creased  emphasis  as  an  important  element  of  systems  affordability.  This  emphasis 
is  evident  within  the  Department  of  Defense,  in  the  Congress, '  and  in  the  General 
Accounting  Office.- 

Investigation  into  the  relationship  between  production  rate  and  production 
cost  for  defense  systems  indicates  significant  variations  in  cost  as  a  function  of 
production  rate.  A  model  that  relates  production  rate,  unit  cost,  and  cumulative 
quantity  has  been  developed.  It  provides  planning  information  of  sufficient 
accuracy  to  assist  in  the  selection  of  economical  production  rates,  and  can  be 
applied  to  a  wide  range  of  complex  systems. 

Department  of  Defense  Instruction  5000.2,  "Major  System  Acquisition  Pro¬ 
cedures,"'  requires  that  production  rate/cost  relationships  be  addressed  at  pro¬ 
gram  Milestone  II  (full-scale  development)  and  at  Milestone  111  (production  and 
deployment).  This  requirement  establishes  the  need  for  a  method  to  explore  these 
relationships. 

A  review  of  the  June  30,  1980,  selected  acquisition  reports'  for  the  46  current 
major  weapon  systems  indicated  that  a  $19.1  billion  cost  growth  h  resulted 
from  schedule  changes  on  these  programs,  which  proves  that  produ.  .ion  rates 
have  a  significant  influence  on  the  cost  of  defense  systems.  Data  compiled  by  the 
General  Accounting  Office  show  that  the  production  rates  for  fighter  aircraft 
have  declined  from  600-700  per  year  in  the  1950s  to  less  than  150  per  year  in  the 
1980s. 

In  this  article,  I  propose  one  method  (or  estimating  rate  cost  quantity  rela¬ 
tionships  using  cost  estimates  or  empirical  data  points  which  are  unique  to  each 
system.  Methodology  for  performing  the  analysis  is  explained,  and  an  example  is 
given.  Arithmetic  operations  were  performed  using  a  time-sharing  computer  with 
its  available  routines  for  regression  analysis.  The  current  generation  ot  hand-held 
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calculators  is  also  capable  of  making  the  calculations.  In  this  article,  unit  cost  is 
defined  as  cost  to  the  customer. 

Rate /Cost/ Quantity  Model 

Traditionally,  system  unit  costs  have  been  projected  using  the  familiar  ex¬ 
perience  cui've.  This  method  depicts  the  projected  unit  cost  as  a  function  of 
cumulative  quantity  produced,  without  regard  for  the  production  rates  involved. 
Variations  in  unit  cost  as  a  function  of  production  rate  appear  to  be  largely  due  to 
amortization  of  fixed  overhead.  The  rate/cost/quantity  model  discussed  here 
adds  production  rate  as  the  "Z"  axis  of  a  three-dimensional  response  surface.  This 
model  is  based  on  the  model  developed  as  a  result  of  investigations  into  the  rela¬ 
tionship  between  direct  labor  requirements  and  production  rate  by  L.  L.  Smith* 
and  further  verified  by  others. ? 

Data  inputs  for  this  model  consist  of  historical  rate/cost/quantity  data  for 
ongoing  programs,  and  contractor  or  in-house  estimates  for  new  programs.  Only 
unit-fly-away  (roll-away,  swim-away,  etc.)  costs  are  considered,  not  total  pro¬ 
gram  costs.  A  multiple  regression  is  performed  in  which  unit  cost  is  the  dependent 
variable,  and  cumulative  quantity  and  production  rate  are  the  independent 
variables.  By  this  method,  an  equation  is  derived  which  describes  the  three- 
dimensional  response  surface  relating,  these  variables.  Most  of  the  sets  of  data 
analyzed  in  this  manner  to  date  show  a  high  multiple  correlation  coefficient  (in 
excess  of  0.9).  Figure  1  depicts  this  response  surface  and  the  relationship  between 
the  variables. 

Figure  2  depicts  the  response  surface  for  a  hypothetical  weapon  system  and 
will  serve  to  compare  the  experience  curve  method  with  the  rate/cost/quantity 
method. 

Given: 

—First  unit  cost  =  $1000  for  a  rate  of  bOO  units/ year 
—90  percent  experience  curve  slope  (exponent  -  .152) 

—85  percent  rate  cost  curve  slope  (exponent  -  .235) 

—6000  units  to  be  produced 


4  Larry  Lacross  Smith.  An  Investigation  of  Changes  in  Direct  Labor  Requirements  Resulting 
from  Changes  in  Airframe  Production  Rate.  Eugene,  Oregon:  University  of  Oregon.  June  1976.  Ph.D. 
Dissertation. 

5  Duane  E.  Conglelon  and  David  W.  Kinton,  An  Empirical  Study  of  the  Impact  of  a  Production 
Kate  Change  on  the  Direct  Labor  Requirements  for  An  Airframe  Manufacturing  Program.  Wright- 
I’atterson  Air  Force  Base.  Ohio;  Air  Force  Institute  ot  Technology.  September  1977. 
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FIGURE  1 

Response  Surface  Relating  Unit  Cost, 
Cumulative  Quantity,  and  Production  Rate 


Experience  curve  method: 

(1)  Y  =  AX^  =  1000 

Rate/cost/quantily  method: 

Y  =  (K)  Quantity'  Rate' 

Using  first  unit  cost  data  point  to  solve  for  K: 

1000  =  (#0  ^00 
K  =  4482 

Therefore: 

(2)  Y  =  4482 

When  equations  (1)  and  (2)  are  used  to  project  unit  costs  vs.  cumulative  quantity  at  a 
constant  production  rate  of  600  per  year,  the  results  are  identical.  When  they  are 
used  to  predict  unit  costs  for  a  variable  production  rate,  the  results  are  shown  in 
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FIGURE  2 

Comparison  of  Experience  Curve  with  Rate/Cost/Quantity  Method 


Figure  3.  As  can  be  seen  from  this  figure,  the  lower  unit  costs  are  associated  with 
the  higher  production  rates,  and  the  higher  unit  costs  are  associated  with  the 
lower  production  rates 


Case  Study 

The  following  example  of  rate/cost/quantity  relationships  is  based  on  data 
from  an  actual  weapon  system.  The  data  have  been  masked  by  changing  the  first 
unit  cost  and  by  using  a  different  region  of  the  response  surface  in  terms  of  pro¬ 
duction  rates  and  quantities  produced.  The  exponents  for  quantity  and  rate,  as 
well  as  all  the  correlation  coefficients  are  the  actual  values  resulting  from  the 
analysis.  Input  data  for  the  analysis  consisted  of: 

—Annual  quantities  produced  (rate) 

—Average  unit  cost  by  year 
—Cumulative  lot  mid-points  for  each  year 

After  these  data  were  entered  into  the  computer,  the  following  calculations  were 
performed: 


FIGURE  3 

Comparison  of  Predicted  Unit  Costs 


CUMULATIVE  QUANTITY 
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— A  log  transform  was  made  for  each  variable 

— A  linear  multiple  regression  was  performed  in  which  unit  cost  was  the  depen¬ 
dent  variable,  and  production  rate  and  cumulative  quantity  were  the  independent 
variables. 

—Independent  regression  analyses  were  performed  for  rate  vs.  cost  and  quantity 
vs.  cost. 

The  results  of  these  analyses  were  as  follows: 

—The  equation  for  the  response  surface  describing  the  relationship  between  unit 
cost,  production  rate,  and  cumulative  quantity  was  determined  to  be: 

-Unit  cost  ($K)  =  500  X  Quantity®- 1 x  Rate"®-1135 
—Multiple  correlation  coefficient  (R)  “  0.966 
—Results  of  the  individual  regression  analyses: 

-Unit  cost  vs.  quantity  R  =  -0.908 
-Unit  cost  vs.  Production  Rate  R  —  -0.712 
—Slope  of  the  quantity/cost  line  (experience  curve)  =  88  percent 
—Slope  of  the  rate/cost  line  =  92.4  percent 

These  results  were  used  to  describe  a  rate/cost/quantity  response  surface  similar 
to  the  one  depicted  in  Figure  1. 

When  viewed  from  three  separate  viewpoints,  some  interesting  relationships 
between  unit  cost,  production  rate,  and  cumulative  quantity  are  revealed.  Figure 
4  depicts  a  view  from  the  side  of  the  response  surface  in  which  unit  cost  is  the  ver¬ 
tical  axis,  and  cumulative  quantity  is  the  horizontal  axis.  From  this  vantage 
point,  production  rates  of  interest  are  displayed  as  a  family  of  curves.  In  this  ex¬ 
ample,  production  rates  of  96,  180,  and  300  units  per  year  are  shown.  Figure  5 
depicts  a  view  from  the  end  of  the  response  surface  in  which  unit  cost  is  the  ver¬ 
tical  axis  and  production  rate  the  horizontal  axis.  Cumulative  production  quan¬ 
tities  of  300,  600,  900,  and  1,200  are  shown  as  a  family  of  curves.  When  horizon¬ 
tal  "slices"  are  made  through  the  response  surface,  thus  depicting  constant  unit 
costs,  and  the  response  surface  is  viewed  from  the  top,  the  results  are  shown  in 
Figure  6.  In  this  view  production  rate  is  the  vertical  axis  and  cumulative  quantity 
the  horizontal  axis.  The  iso-unit  cost  lines  represent  instantaneous  values  of  unit 
cost  and  illustrate  the  wide  variation  in  unit  costs  which  may  be  expterienced  as  a 
function  of  production  rate  and  cumulative  quantity  produced. 

Together,  these  views  of  the  production  rate,  cumulative  quantity,  unit-cost 
response  surface,  are  of  considerable  value  in  responding  to  the  "what  if"  ques¬ 
tions  which  arise  during  the  planning  and  budgeting  cycles  for  each  system. 

A  cumulative  cost  response  surface  can  also  be  constructed.  This  surface 
depicts  total  cost  as  the  vertical  axis  and  quantity  produced  as  the  horizontal  axis. 
The  family  of  curves  in  this  case  is  production  rate.  Discussion  of  this  analysis  is 
beyond  the  scope  of  this  paper. 
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Cautionary  Notes 

The  use  of  the  rate/cost  quantity  response  surface  is  useful  in  displaying  rela¬ 
tionships  between  these  variables;  however,  some  words  of  caution  should  be  ex¬ 
pressed.  When  using  this  technique,  the  following  should  be  kept  in  mind; 

— Since  the  multiple  correlation  is  made  using  transformed  data,  the  resulting 
correlation  coefficients  apply  to  the  transformed  data  rather  than  the  original 
rate/cost/quantity  data.  Correlation  coefficients  for  the  original  input  data  may 
be  larger  or  smaller  than  those  of  the  transformed  data. 

— For  planning  purposes  it  may  be  necessary  to  extrapolate  values  outside  the 
range  of  the  input  data.  This  should  be  done  with  extreme  caution,  with  due  con¬ 
sideration  being  given  to  the  other  variables  which  may  affect  rate/cost/quantity 
relationships.  Among  these  variables  are  the  capacity  of  existing  tooling  and 
facilities,  availability  of  manpower,  ability  of  major  subcontractors  to  supply  the 
required  sub-systems  and  materials,  and  the  effects  on  other  concurrent  programs. 
—Since  production  rate  and  cumulative  quantity  are  not  truly  independent 
variables,  in  some  cases  a  high  degree  of  mulficolinearity  may  exist.  This  may 
result  in  a  change  in  sign  for  one  of  the  variables  in  the  equation  for  the  response 
surface.  This  can  be  corrected  by  the  use  of  "dummy"  data  points  derived  from 
actual  data  and  the  results  of  the  individual  regression  analyses.  This  remedy, 
however,  adds  uncertainty  to  the  overall  results. 

—Results  from  the  analysis  of  several  systems  indicate  that  a  wide  variability 
exists  in  the  slopes  of  the  quantity/cost  lines  and  the  rate/cost  lines.  Use  of  in¬ 
dustry  average  or  commodity  average  slopes  may  introduce  substantial  error  into 
rate/cost/quantity  projections. 

Summary 

Although  production  rate  is  just  one  of  the  important  variables  to  be  con¬ 
sidered  in  the  acquisition  of  defense  systems,  it  is  important  in  terms  of  unit  cost. 
Other  important  variables,  of  course,  would  be  military  requirements,  contractor 
tooling  capacity,  ability  of  subcontractors  to  supply  components,  operation  and 
support  considerations,  etc. 

The  model  presented  in  this  paper  uses  constant  dollars  throughout.  In 
addition  to  the  difference  in  program  cost  in  terms  of  constant  dollars,  additional 
costs  are  incurred  through  a  longer  exposure  to  inflation.  Of  the  $19.1  billion  for 
schedule  changes  on  the  current  SAR  programs,  approximately  one-third  are  con¬ 
stant  dollars  and  two-thirds  are  due  to  inflation.  It  is  difficult  to  annualize  these 
figures  because  some  of  the  systems  have  been  on  line  for  a  number  of  years,  and 
others  are  just  going  into  production.  In  addition  to  the  fact  that  production  rate 
vs.  production  cost  consiHorations  are  required  to  be  made  at  DSARC  II  and  III, 
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use  of  the  rate/cost/quantity  model  can  be  invaluable  in  e  loring  the  inevitable 
"what  if"  questions  that  arise  during  the  planning  and  budgeting  cycles.  Graphic 
displays  of  the  rate/cost/quantity  relationships  allow  rapid  approximations  of 
the  effects  of  incremental  funding  changes  on  the  specific  program  as  well  as  the 
effects  of  changes  in  quantity  procured.  Other  questions  that  can  be  approx¬ 
imated  from  the  graphic  displays  are  the  cost  effects  of  program  stretchouts,  costs 
of  maintaining  a  warm  production  base,  and  the  probable  effects  of  program 
acceleration.  || 
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The  VAMOSC  Connection: 
Improving  Operating  and 
Support  Costing 

Alvin  M.  Frager 


1  he  increasing  complexity  and  unit  cost  of  weapon  systems  have  resulted  in 
significant  increases  in  acquisition  costs.  At  the  same  time,  these  increasing  com¬ 
plexities  have  resulted  in  less  reliable  systems  that  are  more  difficult  to  maintain. 

Increases  in  operations,  maintenance,  and  military  personnel  accounts  have 
not  been  due  solely  to  equipment  maintenance  problems.  Recurring  manpower 
shortages  have  compounded  maintainability  problems.  When  these  are  joined 
with  the  significant  increases  in  energy  costs,  O&S  costs  increase  drastically. 

The  priorities  of  defense  systems  acquisition  managers  are  generally  focused 
towards  on-schedule  development  and  deployment  of  weapons  that  meet  specific 
capabilities.  However,  Jacques  Gansler,  in  his  book.  The  Defense  Industry, 
restates  the  generally  accepted  notion  that  "the  extreme  R&D  emphasis  on  equip¬ 
ment  performance  has  rapidly  increased  the  cost  of  military  equipment  and  has 
thus  greatly  reduced  the  quantities  of  equipment  procured.  Performance  has  been 
increased,  but  at  too  high  a  cost."' 

In  the  current  environment  of  constrained  budgets,  defense  buys  fewer,  more 
expensive  weapons.  Concurrently,  force  readiness  suffers  because  of  inadequate 
logistics  support. 

The  recent  presidential  campaign  addressed  this  problem  and  its  impact  on 
defense  readiness  in  the  debate  that  focused  on  the  need  for  increased  defense 
funding  and  better  defense  management.  The  recognition  of  support  cost  and 
reacMness  concerns  has  started  to  produce  greater  understanding  of  trade-offs 
neceisary  to  control  total  cost. 

Design-to-cost  initiatives  continue  to  focus  management  attention  on  pro¬ 
gram  producibility.  They  motivate  the  services  and  industry  to  search  for  lower 
cost  c  esign  alternatives  to  satisfy  mission  requirements.  DOD  Directive  5000.28, 
released  in  May  1975,  stated  that;  "Design-To-Cost  is  a  management  concept 
wherein  rigorous  cost  goals  are  established  during  development  and  the  control 
of  systems  costs  (acquisition,  operating,  and  support)  to  these  goals  is  achieved 
by  practical  tradeoffs  between  operational  capability,  performance,  cost,  and 
schedule.  Cost,  as  a  key  design  parameter,  is  addressed  on  a  continuing  basis  and 
as  an  nherent  part  of  the  development  and  production  process."  This  directive 
forma  <zes  DOD's  intention  to  manage  defense  system  life-cycle  costs  (LCC).- 


1.  See  J.  S.  Gansler.  The  Defense  Jmiustry.  The  MIT  Press  1*^80. 

2.  The  life-cycle  cosl  of  a  system  is  the  total  cost  to  the  government  of  acquisition  and  ownership 
of  that  system  over  its  full  life.  It  includes  the  cost  of  development,  acquisition,  operation,  support, 
and  where  applicable,  disposal,  DODD  5000.28.  May  1975. 


A/rin  M.  Frager  is  Manager  of  the  Life  Cyc/e  Cost  and  Readiness  Analysis  Department  at  Inforrna- 
tion  Spectrum.  Inc-  He  has  sensed  on  the  staff  of  the  Assistant  Secrefflrv  of  Defense  f Manpou'er 
Reser\'e  Affairs,  and  Logistics).  Directorate  for  Acquisition  and  Support  Planning  Mr  Frager  holds  *4 
B  S  degree  in  electrical  engineering  from  the  University  of  Mis.'jour: 


96 


Concepts 


Clearly,  DODD  5000.28  generates  designer  emphasis  on  life-cycle  cost,  but  in 
doing  so  it  recognizes  that  the  key  problem  associated  with  LCC  management  and 
reduction  is  the  ability  to  estimate  the  O&S  cost  of  weapon  systems:  "Because  of 
the  ability  to  more  accurately  estimate  production  costs  and  the  supportive  pro¬ 
duction  cost  data  base  available,  initial  goals  for  Design-To-Cost  shall  be 
established  in  the  form  of  Average  Unit  Flyaway  (Rollaway,  Sailaway)  cost.  Pro¬ 
grams  to  strengthen  the  data  base  of  weapon  system  operation  and  support 
(O&S)  cost  shall  continue.  As  the  ability  to  translate  O&S  cost  elements  into 
design  requirements  improves,  Design-To-Cost  goals  may  be  extended  into  this 
area.” 

Consistent  with  this  guidance,  DOD  defined  a  management  by  objective 
initiative  9-2,  committing  DOD  to  "define  and  establish  a  management  informa¬ 
tion  system  which  will  provide  data  on  maintenance  and  operation  cost  by 
weapon  system."  Each  of  the  services  responded  to  the  MBO  by  efforts  directed 
towards  development  of  a  prototype  O&S  cost  information  system.  Within  the 
acquisition  process,  VAMOSC  reports  are  intended  to  provide  explicit  O&S  cost- 
related  data,  consistent  with  Department  of  Defense  O&S  costing  guidance 
outlined  several  times  by  the  Office  of  the  Secretary  of  Defense,  Cost  Analysis 
Improvement  Group  (CAIG). 

The  VAMOSC  goal  is  to  provide  O&S  cost  reports  for  each  major  weapon 
type  and  series.  At  this  time.  Navy  VAMOSC  reports  are  available  for  4  years  for 
aircraft,  and  for  2  years  for  most  ships.  Air  Force  VAMOSC  reports  have  been 
developed  for  all  aircraft  by  an  initial  system,  and  the  Air  Force  has  a  major  revi¬ 
sion  currently  underway  that  will  be  operational  in  several  years.  VAMOSC 
reports  for  other  Air  Force  weapon  systems  are  expected  to  follow.  Army 
VAMOSC  products  are  under  design  and  development. 

VAMOSC  data  are  being  developed  in  two  basic  costing  formats.  The  first 
provides  data  for  each  weapon  type  in  a  "top  level"  cost  element  structure  (CES). 
An  abbreviated  example  of  the  Navy  report  for  air  systems  is  shown  in  Figure  1, 
and  for  ships  in  Figure  2.  The  cost  data  provided  in  these  top-level  VAMOSC 
reports  reflect  system  demands  and  material  consumption. 

The  second  type  of  VAMOSC  report  provides  maintenance-related  cost  data 
at  a  detailed  level.  Figure  3  is  an  illustration  of  this  report  for  Navy  aircraft.  This 
report  provides  designers  and  analysts  with  information  identifying  hardware 
and  logistics-related  cost  drivers.  A  report  is  available  that  identifies  comparable 
costs  for  ship  equipment.  The  detailed  cost  data  in  Figure  3  are  summarized  to 
each  cost  element  and  to  the  total  system. 

With  these  products,  the  system  designer  and  cost  analyst  have  the  necessary 
data  to  improve  the  approaches  to  estimate  O&S  costs  of  alternatives  at  all  hard¬ 
ware  levels,  and  to  provide  a  basis  in  experience  to  comment  on  the  cost  impact 
of  proposed  system  changes.  An  evaluation  of  a  proposed  system  that  considers 
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design  and  logistic  support  differences  between  alternatives  can  be  the  basis  of 
predictions  of  O&S  cost  differences.  The  differences  in  operating  scenarios  can  be 
equated  into  O&S  cost  increments.  When  calculated  costs  are  related  back  to  the 
O&S  cost  reported  by  VAMOSC,  the  estimates  for  the  new  system  can  be  directly 
compared  to  O&S  cost-experience  data  for  existing  systems.  An  engineering- 
based  costing  technique  such  as  this  provides  estimates  that  are  credible,  and 
therefore  can  be  expected  to  be  more  acceptable  to  the  decision-maker.  Clearly, 
the  approach  taken  by  the  user  of  VAMOSC  data  will  be  a  critical  element  affect- 
iirg  the  credibility  of  the  estimates.  With  the  detail  and  visibility  provided  by 
VAMOSC  reports,  the  analyst  can  use  existing  costing  methodology  to  develop 
improved  O&S  estimates. 

The  importance  of  cost  management  in  new  systems  acquisition  has  already 
been  reiterated  in  the  new  administration.  In  a  memorandum  distributed  within 
DOD,  the  Deputy  Secretary  of  Defense  stated  the  priority  objective  to  "reduce 
costs  by  looking  for  substantial  and  real  savings  in  the  acquisition  of  major 
weapons  systems;  .  .  .’ 

It  is  important  that  the  systems  design  community  increase  their  use  of 
VAMOSC  information.  The  designer  must  be  able  to  recognize  the  impact  on 
cost  resulting  from  parameters  that  he  can  directly  control.  The  designer  may 
make  a  choice  between  a  hydraulic  or  electric  actuator.  The  program  manager 
must  choose  between  various  material  mixes.  The  Defense  Systems  Acquisition 
Review  Council  reviewers  must  understand  the  O&S  cost  impact  of  a  degradation 
in  system  performance  parameters.  The  cost  variable  in  a  decision  needs  to  be 
described  in  terms  that  can  be  equated  into  specific  design  alternatives.  In  a  gross 
sense,  empty  weight  or  engine  thrust  are  parameters  that  a  designer  wants  to  con¬ 
trol.  These,  however,  are  difficult  for  a  designer  to  deal  with  in  the  specific  trade¬ 
off  studies  where  the  design  alternatives  affect  reliability  of  a  component,  its 
repair  time,  cost,  material  mix,  alternative  engine  combustion  cycles,  location  of 
access  doors,  and  avionics  level  of  modularity.  The  O&S  cost  factors  must  be  ex¬ 
plicitly  related  to  design  and  support  characteristics  of  hardware. 

The  most  explicit  approach  for  estimating  an  O&S  cost  factor  commonly  is 
referred  to  as  "bottoms-up,"  or  as  an  engineering  build-up  approach,  illustrated 
in  Figure  4.  The  estimate  is  "built  up"  by  aggregating  the  contributions  to  the  total 
cost  made  by  significant  system  functional  hardware  items.  In  this  case,  the  im¬ 
pact  of  each  design-related  input  factor  such  as  replenishment  spares,  component 
rework,  etc.,  is  built  up  by  aggregating  all  of  the  contributions  by  hardware  it  c..n 
to  that  cost  factor. 

3.  See  Memorandum  for  Secretaries  from  the  Deputy  Secretary  of  Defense  to  major  DOD  offices, 
"Improving  The  Defense  Acquisition  System  and  Reducing  System  Costs,"  2  March  1981. 
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FIGURE  4 

F/A-XX  Cost  Factor  Buildup 
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The  VAMOSC  maintenance-related  cost  data  can  provide  the  needed  detail 
for  an  analyst  to  build  up  the  O&S  cost  estimate  explicitly  from  design  and 
logistics-related  info.mation.  It  can  provide  current  experience  data  for  existing, 
installed  components  that  comprise  the  system.  Typically,  hardware  in  a  weapon 
system  evolves  towards  improved  performance,  and  the  general  functions  of  the 
hardware  will  evolve  from  one  system  to  its  follow-on  system.  As  such,  it  is  to  be 
expected  that  there  is  correlation  of  cost  experience  between  current  and  advanced 
generation  equipment. 

With  VAMOSC,  cost-data  analogies  can  be  identified,  or  scaling  techniques 
can  be  employed,  to  consider  the  impact  of  current  operating  hardware  ex¬ 
perience  to  advanced  hardware  operating  in  an  equivalent  environment.  Where 
the  system  or  function  is  unchanged,  and  the  operating  environment  is 
equivalent,  the  existing  VAMOSC  data  for  the  system  or  component  may  be 
assumed  to  be  analogous  to  the  anticipated  experience  for  new  system  or  compo¬ 
nent.  In  cases  where  equipment  technology  has  advanced  to  the  point  where  ex¬ 
isting  experience  data  cannot  be  considered  directly  analogous,  the  analyst  is  still 
better  off  with  VAMOSC  data  because  the  estimate's  uncertainty,  due  to  lack  of 
experience  data,  is  limited  to  that  specific  hardware  element  of  the  total  estimate. 
A  bottoms-up  estimate  of  key  O&S  cost  factors,  using  a  scaling  approach,  is  il¬ 
lustrated  in  Figure  5.  In  this  example,  reliability  and  material  cost  factors  have 
been  developed  that  relate  analogous  equipment  to  new  design  equipment.  Based 
on  these  scalars,  an  estimate  has  been  developed  for  an  advanced  aircraft  that  can 
be  compared  to  VAMOSC  experience  data  for  existing  operational  (e.g.,  bench¬ 
mark)  aircraft. 

The  bottoms-up  costing  approaches  can  be  improved  with  the  use  of  contrac¬ 
tor  logistic  support  analyses  (LSA),  or  by  data  developed  in  comparability 
analyses  that  identify  analogies  or  "like  or  similar"  functions  to  existing  hard¬ 
ware.  During  full  scale  development,  cost  estimates  by  hardware  contractors  for 
design  trade-off  analyses  are  more  likely  to  use  LSA  data  because  such  efforts  are 
increasingly  required  as  an  integral  part  of  the  contractor  design  and  integrated 
logistics  support  (ILS)  planning  process. 

Specific  approaches  to  O&S  cost-factor  estimation  to  the  exclusion  of  others 
should  be  avoided  because  no  single  approach  can  be  expected  to  credibly 
estimate  costs  in  all  situations.  At  any  stage  of  a  program's  acquisition  and  design 
process,  the  specific  hardware  elements  may  be  at  different  stages  of  definition 
and  production,  as  well  as  being  a  mix  of  off-the-shelf  and  new-design  hardware. 
The  availability  of  VAMOSC  data  by  weapon-system  type  and  series  allows  an 
analyst  greater  flexibility  to  define  and  implement  an  estimating  approach  that 
will  provide  a  more  explicit  O&S  cost  estimate.  The  cost  estimate  will  be  more 
credible  to  management  because  the  estimates  can  be  related  back  to  a  com¬ 
parable  baseline  system.  This  relationship  will  provide  the  program  manager 
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with  the  capability  to  determine,  in  design-  and  logistics-related  terms,  the  an¬ 
ticipated  differences  in  O&S  cost  between  the  new  and  an  existing  system. 

It  is  this  attribute  of  O&S  cost  estimates  that  has  been  missing  in  the  past.  The 
VAMOSC  data  provides  the  needed  credibility  to  convince  the  program  manager 
that  the  calculated  O&S  cost  impacts  of  program  alternatives  are  reasonable,  and 
therefore  deserve  consideration  in  the  decision  process.  || 
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Is  Cost  Growth 
Being  Reinforced? 

Noreen  S.  Bryan 
Dr.  Rolf  Clark 


Cost  growth  is  a  major  problem  in  defense  systems  acquisition.  In  recent 
congressional  hearings.  Chairman  Jack  Brooks  stated:  "Since  1969  the  DOD  has 
underestimated  costs  of  all  major  weapon  systems  by  more  than  50  percent."'  Yet 
this  is  only  half  of  the  cost  growth  problem.  What  Chairman  Brooks'  committee 
was  investigating  was  growth  within  a  program — from  the  time  of  its  initial  plan¬ 
ning  estimate  to  actual  delivery  of  production  systems.  There  is  another  aspect  of 
cost  growth,  namely  the  trend  of  increasing  unit  costs  from  one  generation  of 
systems  to  the  next.  We  will  need  to  deal  with  both  of  these  aspects. 

No  single  factor  can  be  identified  as  the  cause  for  the  increased  cost  of  military 
systems.  Investigations  of  cost  growth  have  identified  inflation,  technical 
changes,  quantity  decreases,  overoptimism  and  "buy-ins,"  and  reduced  DOD 
budgets  as  major  causes  of  cost  growth  and  resultant  increased  unit  cost  of 
systems.  Program  management  has  been  criticized  for  its  lack  of  adequate  control 
of  contractors.  Yet,  even  in  aggregate,  these  well-known  causes  do  not  provide  a 
complete  picture. 

There  are  also  policy  and  management  factors  that  are  significant  con¬ 
tributors  to  cost  growth.  In  the  hearings  referenced,  Jerome  Stolarow  hinted  at 
some  of  the  policy  issues  causing  growth,  but  they  were  never  developed  beyond 
an  implication  that  inflation,  complexity,  and  other  factors  were  being  used 
merely  as  scapegoats  to  explain  cost  overruns.  In  this  paper  we  will  try  to  explain 
some  of  these  complex  policy  and  management  issues  that  are  not  well 
understood.  We  leave  the  more  traditional  (but  also  valid)  cost  growth  explana¬ 
tions  for  others  to  present. 


1.  "Inaccuracy  of  Department  of  Defense  Weapons  Acquisition  Cost  Estimates.*'  Hearings  before 
a  subcommittee  of  the  Committee  on  Government  Op>erations,  96th  Cong.,  1st  sess.,  June  25  and  26, 
1979,  p.  2. 
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II 


Interdependence  of  Actual  Costs  for  Old  Systems 
and  Cost  Estimates  for  New  Systems 

The  history  of  cost  growth  for  individual  programs  is  abundant.  Case 
histories  for  18  Navy  systems  are  provided  in  the  Brooks  hearings  which  show 
how  much  current  system  costs  have  risen  above  the  initital  estimates.  But  there 
is  also  the  question  of  increased  unit  costs  from  one  system  to  another— from  the 
F-14  to  the  F-18  aircraft,  for  example,  or  from  the  Knox-class  frigate  to  the  Perry 
class.  Review  of  unit  costs  shows  a  pattern  that  successor  systems  are  indeed 
costing  more.  Table  I  shows  examples  of  cost  growth  over  time  for  acquisition  of 
selected  major  systems. 

Review  of  these  examples  shows  increases  in  unit  costs  even  after  adjusting  for  in¬ 
flation  and  weight.  What  are  some  of  the  causes,  valid  and  not-so-valid,  for  such 
growth?  First  it  is  necessary  to  look  at  the  methods  for  cost  estimating.  Common 
to  all  methods  is  the  use  of  actual  historical  costs  as  the  basis  for  projecting  costs 
for  a  new  system.  Implicit  in  all  the  techniques  is  the  assumption  that  what  has 
occurred  in  the  past  is  a  good  guide  to  what  will  occur  in  the  future.  The  esulting 
forecasts  of  what  will  happen  is  based  on  the  assumption  that  the  same  forces  as 
have  been  operating  in  the  past  will  continue  to  operate  in  the  future.  Thus,  the 
cost  trends  we  saw  in  unit  cost  histories  for  aircraft  and  ships  in  Table  1  are  in¬ 
tegral  to  the  cost-estimating  relationships  and,  inevitably,  to  expectations  of  the 


TABLE  I 

Unit  Costs,  Adjusted  for  Inflation  and  for  Size 


Fighter  Aircraft  at  Unit  200 

Attack  Aircraft  at  Unit  200 

Frigate-Type  Lead  Ships 

(1981  $  per  lb.) 

(1981  $  per  lb.) 

(1981  $  per  KTon) 

year 

Model 

Cost 

Year 

Model 

Cost 

Year 

Class 

Cost 

1956 

F-8 

ISl 

1953 

A-4 

$195 

1963 

Garcia 

$11,500 

1962 

F-4 

1959 

A-6* 

$425 

1968 

Knos 

$18,500 

1970 

F-14 

$480 

1965 

A-7 

1978 

Perryb 

$35,000 

1980 

F/A.18 

$700 

1982 

FIA-18 

_ 

Notes;  a.  A-6  production  rate  was  low  (about  30  per  year  vs.  120  for  the  A-4 
and  60  for  the  A-7]  causing  unit  costs  to  be  high  comparatively. 


b.  The  Perry-class  frigate,  being  a  guided-missile  ship  has  been 
downward  adjusted  (based  on  fleet-wide  data)  for  comparability  with 
the  Garcia  and  Knos,  both  noivmissile  ships. 
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costs  of  future  systems.  In  fact,  since  the  complexity  of  most  of  our  military 
systems  is  continually  increasing,  one  can  conclude  that  the  trend  of  increasing 
unit  costs  over  time  will  continue  in  the  future. 

The  Role  of  Increased  "Ilities" 

Disaggregation  of  the  unit  costs  of  systems  into  their  components  indicates 
that  much  of  the  unit  cost  increase  is  attributable  to  new  "ilities”  and  management 
support  requirements.  Acquisition  funds  for  fielded  systems  include  not  only  the 
deliverable  hardware  units,  but  all  of  the  support  items — tests,  engineering 
analyses,  management  functions,  etc. — required  for  operational  systems.  Spares 
kits,  training,  and  documentation  are  well-known  examples  of  support  items 
which  are  tangible  deliverables  like  the  weapon  system  itself.  But  there  are  addi¬ 
tional  support  requirements  that  are  levied  on  programs  with  outputs  that  affect 
the  quality  of  a  system,  but  which  are  not  products  in  themselves.  These  support 
requirements  include  configuration  management,  integrated  logistics  support, 
habitability,  system  safety,  human  factors  engineering,  reliability,  maintainabili¬ 
ty,  design-to-cost  and  life-cycle-cost  requirements,  and  an  ever-growing  list  of 
others.  These  management  support  requirements  have  come  to  be  known  com¬ 
monly  as  the  "ilities." 

As  each  of  these  "ilities"  is  added  to  the  specification  for  a  new  system,  the 
prime  contractor  assembles  a  team  to  perform  the  task.  In  addition,  the  govern¬ 
ment  adds  documentation  requirements  for  plans,  programs,  technical  reports 
and  test  reports.  The  contractor  then  adds  the  additional  technical  publication 
editors  and  illustrators  to  fulfill  these  requirements.  The  net  result  is  the  addition 
of  a  new  staff  of  people.  Further,  the  support  of  those  support  people  must  be 
coordinated  into  the  design  and  fabrication  of  systems:  added  management 
responsibilities  and  personnel,  and  certainly  added  costs,  are  the  result.  Ex¬ 
perience  in  evaluating  proposals  and  return  costs  for  Navy  programs  has  shown 
that  for  each  new  support  discipline,  the  addition  of  at  least  one  person  to  the 
management  team  is  required.  Those  teams  that  were  once  limited  to  a  program 
manager  and  his  assistants  in  financial  and  contractual  management  have  now 
expanded  to  become  unwieldy  teams  including  10  to  15  additional  disciplines. 
The  result  of  these  additions  of  disciplines  and  people  is  to  partition  an  ever- 
larger  proportion  of  procurement  dollars  away  from  the  weapon  units  delivered. 
Attacking  this  problem  is  particularly  difficult  because,  while  each  discipline  by 
itself  adds  only  a  small  percentage  of  cost  to  a  system,  the  aggregate  effect  of 
these  added  disciplines  is  significantly  increased  costs. 

The  "ilities"  are  necessary,  or  at  least  highly  desirable.  But  at  what  cost?  We 
question  the  addition  of  these  "ilities"  without  a  system  to  address  their  "value 
added."  Typically,  these  trends  of  increasing  "ilities"  and  support  functions  are 
not  brought  forward  for  review,  but  rather  are  treated  as  necessary  implementa¬ 
tions  of  government  regulations  and  are  built  into  the  cost  of  subsequent  systems. 
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Unlike  in  a  profit-motivated  organization,  the  check  and  balance  system  which 
asks  what  is  the  cost  relative  to  the  value  received  from  each  of  these  added 
disciplines  is  missing. 

Within  the  cost  area  itself,  cost  schedule  control  systems  criteria  (C/SCSC) 
were  established  as  a  method  to  alert  government  management  when  perform¬ 
ance  under  a  specific  contract  was  deviating  significantly  in  cost  or  schedule.  Too 
often,  application  of  C/SCSC  has  resulted  in  an  elaborate  set  of  procedures 
which  may  not  be  worth  their  cost,  particularly  if  the  government  management 
team  is  inadequately  staffed  to  analyze  and  use  the  results.  Typically,  the  im¬ 
plementation  of  C/SCSC  has  resulted  in  the  expansion  of  the  contractor's  finan¬ 
cial  management  team  threefold  to  fivefold.  A  program  that  required  two  finan¬ 
cial  managers  to  track  a  program's  cost  and  schedule  progress  prior  to  C/SCSC 
now  requires  from  six  to  ten  people. 

Presuming  that  this  pattern  of  cost  increases  is  occurring,  the  question  arises 
as  to  why  evidence  of  these  increases  is  not  readily  apparent  in  historical  cost 
data.  The  answer  involves  the  fact  that  most  data  is  aggregated  at  too  high  a  level 
to  make  these  effects  obvious.  Additionally,  estimating  techniques,  particularly 
parametric  CERs,  include  the  costs  of  the  "ilities"  as  undefined  components  of  the 
system  complexity  (i.e.,  physical  or  performance  characteristics).  As  stated 
earlier,  the  complexity  of  most  military  systems  has  been  increasing  with  time. 
Since  the  number  and  dollar  magnitude  of  the  "ilities"  has  also  been  increasing 
with  time,  the  resulting  CERs  are  composites  of  these  two  effects.  Figure  1 
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demonstrates  this  concept.  The  "ilities"  increase  the  upward  trend  of  unit  cost 
that  is  already  increasing  based  on  complexity. 

Program  Cost  Growth — Motivation  to  Increase  Cost  Estimates 

Program  cost  growth,  as  discussed  by  Chairman  Brooks  and  his  committee, 
has  been  a  source  of  continued  criticism  by  the  Congress  and  the  press.  Certainly, 
the  motivation  created  by  this  criticism  is  to  try  to  reduce  cost  growths  in  the 
future.  Cost  estimators  have  responded  by  including  allowances  in  cost  estimates 
to  cover  cost  uncertainties.  These  are  typically  added  as  factors  such  as  an 
"engineering-change"  factor,  a  "weight-growth"  factor,  a  "productivity-loss"  fac¬ 
tor,  an  "allowance-for  modification"  factor,  etc.  The  combined  effect  of  these  fac¬ 
tor  additions  is  an  increase  in  the  initial  cost  estimates.  A  first  reaction  might  be 
that  this  added  cost  is  an  application  of  lessons  learned  from  cost  growth  on 
earlier  programs  and  will  eliminate,  or  at  least  lower,  cost  growth  in  the  future. 
This  is  partly  true— there  will  tend  to  be  less  cost  overrun  above  initial  estimates. 
But  we  will  argue  that  a  concurrent  effect  is  higher  unit  costs  overall;  that  indeed 
an  obverse  cost  policy  has  inadvertently  come  into  existence.  Instead  of  creating 
motivation  to  reduce  unit  costs,  the  attempt  to  eliminate  cost  growth  effectively 
works  to  elevate  the  cost  of  delivered  systems. 

Unit  Cost  Increase  for  an  Amphibious  Ship  Class 

Why  this  is  the  net  result  is  not  obvious.  It  will  be  helpful  to  consider  an  actual 
example  of  how  the  "ilities"  and  cost  uncertainty  factors  are  incorporated  in  a 
cost  estimate.  This  estimate  will  be  compared  to  the  actual  costs  of  the 
predecessor  system,  and  the  impact  on  unit  cost  will  be  evaluated.  Two  am¬ 
phibious  ships  will  be  compared.  In  one  case  it  will  be  the  estimated  costs  for  a 
1981-class  ship;  in  the  other  it  will  be  the  actual  costs  for  the  1965  ship,  which  is 
the  last  comparable  class  built.  The  details  of  the  costs  for  each  class  are  shown  in 
Table  II.  A  study  of  this  table  provides  the  following  insights.  In  total,  the  cost 
estimate  for  the  new  amphibious  ship  is  five  to  six  times  higher  than  the  actual 
costs  incurred  for  the  earlier  ship.  When  these  numbers  are  adjusted  for  weight 
and  inflation  differences,  the  new  ship  has  essentially  doubled  in  cost  (and  this  is 
only  the  estimate,  not  the  actual  cost).  Acknowledging  some  differences  in  the 
two  classes,  this  still  seems  a  large  rate  of  growth,  and  indeed  some  of  the  com¬ 
ponents  in  the  cost  estimate  seem  to  have  grown  exorbitantly.  Construction  plans 
are  up  fourfold  from  the  earlier-class  ship;  government-furnished  electronics 
equipment  up  twofold;  hull,  mechanical,  electrical  equipment  up  threefold. 
Demands  for  system  engineering  and  for  habitability,  previously  implicit  in  the 
basic  construction  costs,  now  account  for  an  additional  20  percent  or  so  of  the 
basic  construction  costs. 

Those  items  already  mentioned— system  engineering,  plans,  government  fur¬ 
nished  equipment,  and  habitability — relate  both  to  increased  requirements  for 
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TABLE  II 

Comparable  Ship  Costs  Adjusted  tor  Size  Differences  (Dollars  in  Millions) 


NOTE:  (R)  =  "Requirement" 

(f)  s  "Uncertainly  factor" 


technical  performance  and  new  requirements  for  "ilities"  and  management  sup¬ 
port  items.  In  fact,  there  are  more  increases  for  "ilities "  than  appear  on  the  sur¬ 
face.  Currently,  more  than  half  of  the  system  engineering  costs  are  typically  the 
sum  of  individual  costs  for  "ilities,"  namely  reliability,  maintainability,  safety, 
standardization,  producibility,  survivabiiity/vulnerability,  human  factors,  etc. 
And  certainly  the  total  costs  for  the  "ilities"  are  not  attributed  to  system  engineer¬ 
ing.  Demonstration  of  the  ship's  ability  to  meet  the  "ilities"  requirements  results 
in  an  expansion  of  the  test  and  evaluation  (T&E)  program.  Referring  to  Table  II, 
T&E  is  another  area  of  cost  that  has  increased  significantly. 

The  categories  of  cost  listed  in  Table  II  also  show  evidence  of  the  "cost  uncer¬ 
tainty"  factors  discussed  previously.  The  "Margin  for  Weight  Growth"  category 
is  a  10  percent  factor  added  for  unforeseen  changes  to  the  ship's  weight.  Another 
category,  the  "Allowance  for  Change  Orders,"  has  become  a  major  cost  driver.  In 
an  attempt  to  understand  the  magnitude  of  cost  uncertainty  factors,  we  have  at¬ 
tempted  to  separate  "requirements"  categories  from  "cost  uncertainty"  ones  (an- 
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notated  accordingly  in  the  laole).  In  total,  the  cost  uncertainty  (actors  account  (or 
17  percent  o(  the  1981  ship  cost.  And  this  cost  is  only  (or  those  categories  that  are 
identi(iable  as  allowance  (actors.  In  addition,  there  is  evidence  that  a  portion  of 
the  increases  in  the  requirements  categories  relate  to  cost  allowances  so  that  “cost 
overruns"  can  be  avoided.  For  example,  system  engineering  is  estimated  using  a 
(actor  chosen  (rom  a  range.  When  this  estimate  was  made,  the  13.5  percent 
system  engineering  multiplier  was  the  maximum  used  by  ship  estimators,  yet  this 
speci(ic  ship  has  apparently  been  touted  as  not  being  particularly  complex.  When 
used  this  way,  the  (actors  can  drive  up  the  unit  cost  o(  ships  in  an  uncontrolled 
way. 

In  an  environment  without  true  management  reserves,  and  one  in  which 
motivation  is  to  have  initial  estimates  match  the  eventual  end  costs  o(  the  system, 
the  use  o(  such  "factors"  becomes  a  necessity,  an  apparent  method  to  hedge 
against  cost  growth.  But,  the  practice  can  be  overdone.  The  (actor  (or  change 
orders  is  one  o(  particular  concern.  In  the  last  IS  years,  and  especially  since  the 
great  claims  litigations  o(  recent  times,  there  has  been  emphasis  to  reduce 
changes.  Ships  are  probably  changed  less  now,  yet  there  is  a  major  allowance  (or 
changes  in  the  1981  ship,  whereas  it  was  minimal  (or  the  earlier  class.  As  a 
percentage  of  the  total  ship  costs,  we  are  now  estimating  changes  to  be  10  percent 
o(  the  total  cost,  while  they  were  2  jjercent  o(  the  previous  class.  Is  that 
reasonable?  Another  allowance  (actor  is  the  15  percent  productivity  loss  in  labor 
e((iciency  that  is  not  shown  but  which  is  (actored  into  the  basic  construction  cost 
o(  the  1981  ship.  This  productivity  loss  occurred  in  the  past  when  shipbuilding 
declined  in  the  United  States.  Should  it  be  gained  back  as  shipbuilding  ac¬ 
celerates?  The  mere  (act  that  productivity  loss  occurred  should  provide  evidence 
that  productivity  gains  may  occur  when  economic  (actors  (eed  back  into  the  ship¬ 
building  industry  to  make  it  less  likely  to  lose  more  productivity.  Is  this  produc¬ 
tivity  loss  "trend"  to  be  incorporated  (orever  in  the  cost  process?  These  are  the 
types  o(  questions  that  policy-makers  must  ask. 

The  intent  here  is  not  to  single  out  problems  in  ship  costs,  but  rather  to  pro¬ 
vide  examples  o(  how  cost  (actors  arise  as  ways  to  compensate  (or  past  induences 
on  growth.  They  occur  in  all  systems.  Nor  can  we  blame  cost  estimators  (or  try¬ 
ing  to  avoid  the  cost  overruns  o(  the  past.  In  (act,  current  policy  motivates  them 
to  make  estimates  high  enough  so  that  serious  cost  growth  in  a  program  will  be 
avoided.  But  the  attempts  are,  we  o((er,  doomed  to  (ail.  For  by  avoiding  cost 
growth  in  a  program,  we  may  be  causing  excessive  growth  in  cost  between  pro¬ 
grams,  the  result  being  an  extension  o(  the  trend  o(  increasing  unit  cost.  We  turn 
to  that  dilemma. 

The  Cost  Spiral— The  Interaction  of 
Cost  Estimates  in  the  Acquisition  Process 

We  have  looked  at  how  (uture  cost  estimates  are  based  on  past  cost  histories 
and  have  seen  examples  of  how  cost  histories  are  incorporated  into  estimating 
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methods  and  support  increased  unit  costs  of  Navy  systems.  We  will  next  consider 
how  these  estimates  are  converted  into  expenditures  and  investigate  how  this 
process  affects  unit  costs. 

The  estimator  projects  the  cost  of  the  new  system  based  on  the  "end  costs"  of 
the  predecessor  system.  "End  cost"  means  that  all  the  costs  associated  with  the 
earlier  system  are  included— not  just  the  costs  envisioned  initially,  but  also  the 
added  costs  accumulated  during  the  research,  development,  and  production 
phases.  These  added  costs  can  be  significant.  Navy  systems  typically  go  through 
iterative  technical  changes,  some  the  result  of  new  operational  requirements, 
others  the  response  to  needed  corrections,  still  others  the  effect  of  technological 
breakthroughs  made  on  subsystems  during  the  acquisition  phase.  Actual  costs  in¬ 
clude  all  these  iterations,  the  scrapping  of  plans,  drawings,  modules  of  hardware 
and  software,  and  the  addition  of  new  plans,  etc. 

Let's  look  at  how  these  estimates  fold  into  the  process  of  acquiring  new 
systems.  Estimates  become  program  budgets,  and  program  budgets  become  funds 
that  government  project  managers  expect  to  expend  to  buy  systems.  If  the  project 
manager,  believing  that  these  funds  represent  probable  "end  costs"  for  the  new 
system,  attempts  to  procure  the  system  within  these  bounds,  then  clearly  he  needs 
to  set  aside  some  amount  of  money  (i.e.,  management  reserve)  to  cover  the 
unknown  but  inevitable  changes.  However,  the  establishment  of  a  management 
reserve  is  contrary  to  prescribed  policy  for  budgeting  and  obligating  funds. 
Regardless  of  the  type  of  funding,  the  program  manager  is  operating  under 
pressure  to  obligate  the  authorized  funds  within  about  12  to  15  months.  But  con¬ 
tractual  regulations  preclude  the  program  manager  from  obligating  money  for 
undefined  tasks;  hence,  the  program  manager  cannot  establish  a  "set-aside " 
budget  or  incorporate  a  management  reserve  in  his  contracts  with  prime  sup¬ 
pliers. ^  The  cost  estimate  may,  as  in  the  ship  case  discussed  above,  set  aside  fac¬ 
tors  for  program  changes,  for  weight  changes,  etc.,  apparently  similar  to 
"reserves."  But  once  a  program  is  authorized  with  these  reserve  factors  included, 
and  once  budgets  are  formed,  the  system  will  be  designed  to  that  entire  authoriza¬ 
tion  number,  and  the  reserves  for  changes  will  be  absorbed  in  the  design.  None 
will  be  left  over  for  unanticipated  changes. 

Consequently,  funding  meant  to  be  adequate  for  the  completed  system,  i.e., 
"end  cost,"  becomes  the  start  cost  instead.  So  system  designers,  both  government 

2.  We  must  be  careful  about  what  is  meant  by  the  term  "management  reserves.”  Management 
reserves  set  aside  by  contractors  as  part  of  the  planning  and  distribution  of  funds  within  their 
organization  are  not  to  be  confused  with  the  government  management  reserves  discussed  in  this 
paper.  The  contractor's  management  reserves  are  used  in  the  performance  of  the  work  that  is  within 
the  scope  of  the  contract.  The  government  management  reserves,  on  the  other  hand,  relate  to  new  re¬ 
quirements  that  are  outside  the  scope  of  the  contract  and  must  be  added  later  through  separate  con¬ 
tractual  actions. 
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and  contractor,  initiate  the  program  by  designing  a  system  at  inception  to  absorb 
the  total  resources  estimated  for  the  completed  system.  Then,  when  undefined 
changes  occur,  their  costs  must  be  added  through  reprogramming  from  other 
sources  (i.e.,  either  other  programs  must  be  sacrificed,  or  quantities  must  be 
reduced).  Both  actions  raise  unit  costs  again,  and  affect  future  unit  costs  as  well. 
When  the  systems  are  finally  delivered  as  production  units,  the  total  actual  costs 
are  once  more  significantly  higher  than  the  original  estimate.  The  conclusion  is 
that  once  again  the  Navy  has  failed  to  adequately  estimate  the  costs.  Certainly,  as 
a  minimum,  the  new  data  point  for  this  system's  cost  must  be  added  to  the  history 
of  data  points  from  previous  systems,  with  the  result  that  the  trend  that  was 
already  increasing  will  be  accelerated  at  a  more  rapid  rate.  In  reality,  the  cost 
growth  problem  is  not  inaccurate  cost  estimating,  but  rather  a  system  that  at¬ 
tempts  to  predict  end  costs,  but  which  does  not  allow  management  to  these  costs. 
Restraining  added  requirements,  acknowledging  that  add-ons  will  occur,  and 
then  having  management  reserves  to  cover  the  necessary  add-ons  are  needed. 

The  cost  spiral  is  one  of  repeating  cycles  of  estimates,  program  budgets  based 
on  the  estimates,  and  contracts  keyed  to  program  budgets.  It  is  a  complex  spiral. 
Estimating  higher  to  acknowledge  past  overruns  seems  reasonable.  Basing 
budgets  on  estimates  seems  reasonable;  designing  to  the  budget  seems  reasonable. 
Yet  because  system  design  is  keyed  to  the  funding  available,  and  because  changes 
are  inevitable,  the  composite  effect  is  inevitable  cost  overrun.  This  cyclical  pat¬ 
tern  is  shown  by  the  flow  diagram  in  Figure  2.  A  quantification  of  one  cycle  is 
shown  in  Figure  3. 

In  order  to  explain  the  concept  of  the  cost  spiral  and  understand  how  it  works, 
it  has  been  necessary  to  simplify  the  acquisition  process  (and,  in  some  ways,  to 
oversimplify  it).  Certainly,  there  are  many  complex  actions  and  policies  that  are 
interacting  concurrently.  For  example,  in  our  description  of  the  cycle,  it  has  been 
presumed  that  the  costs  of  new  systems  are  budgeted  at  the  value  that  is  projected 
by  the  cost-estimating  relationship.  Experience  indicates  that  this  is  not  always 
true.  Other  forces,  such  as  constrained  resources,  or  excessive  optimism  about 
the  facility  of  achieving  technical  requirements,  cause  the  initial  estimates  to  be 
budgeted  below  the  predicted  value.  Referring  to  Figure  3,  the  initial  estimate 
may  be  artificially  optimistic,  and  be  much  closer  to  the  cost  of  the  predecessor 
system  (System  4  on  the  chart).  Despite  this  fact,  however,  the  events  that  follow 
the  initial  budgeting  will  still  occur  as  we  have  described  them,  and  in  fact 
translate  into  similar  proportional  cost  growth.  Systems  are  designed  to  utilize  all 
the  costs  that  are  initially  budgeted.  Changes  and  new  requirements  occurring 
subsequently  still  become  "added  costs"  to  the  program.  The  program  cost 
growth  such  as  that  observed  by  the  Brooks  hearings  will  still  occur— and  the 
motivation  to  avoid  such  cost  overruns  will  be  built  into  future  estimates  once 
again. 
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The  Trade-off  Between  Program  Cost  Overruns 
and  the  Growth  in  Cost  Estimates 

It  would  appear  that  to  restrain  cost  estimates  to  less  than  the  established  cost 
trend  would  only  ensure  larger  cost  overruns  in  the  future.  That  is  true,  but  the 
end  cost  of  systems  might  be  less,  too.  In  other  words,  while  the  cost  growth  be¬ 
tween  a  well-controlled  estimate  and  the  final  end  cost  may  be  higher,  it  is  also 
true  that  the  ultimate  end  cost  of  the  system  could  be  lower  than  if  the  estimate 
simply  followed  the  cost  trend.  The  following  hypothetical  example  applies. 
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In  case  A,  the  growth  in  cost  from  estimate  to  end  cost  is  only  10  percent, 
while  in  B  it  is  50  percent.  Current  policy  makes  case  A  a  more  acceptable  alter¬ 
native  for  the  cost  estimator  and  program  manager.  While  the  system  costs  much 
more,  the  visible  cost  error — that  is,  the  percent  increase  of  end  cost  over  the  in¬ 
itial  cost  estimate — is  much  less.  Under  present  policy,  cost  estimators  and  pro¬ 
gram  managers  will  opt  for  case  A. 

What  Can  Be  Done? 

Awareness  of  the  cost  dynamics,  the  spiral  described,  is  the  first  step.  What  is 
needed  is  a  new  approach  to  cost  policy — one  which  brings  to  bear  the  various 
disciplines  of  the  management  community  to  sort  out  those  categories  of  cost 
which  are  essential  from  those  which  are  allowed  because  estimates  are  based  on 
increasing  CERs  and  trends  rather  than  examination  of  these  trends.  Program 
managers,  contracting  officers,  auditors,  and  especially  the  cost  analysts,  must 
come  to  understand  the  cost  dynamics,  and  how  to  retard  their  growth.  Critical 
review  is  needed  of  the  trend  underlying  the  estimates.  Management  procedures 
which  require  addition  of  support  requirements  without  concern  for  "value  add¬ 
ed"  need  to  be  scrutinized. 
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In  order  to  perform  adequate  cost  control,  the  present  separation  of  the  cost 
analysis  community  from  the  contracting  and  auditing  community  needs  to  be 
eliminated.  Cost  estimators  must  come  to  understand  the  details  of  project 
management,  of  manufacturing,  and  of  contractor  accounting,  so  that  they  can 
isolate  necessary  cost  growth  factors  from  those  that  are  merely  being  trended.  If 
habitability  is  an  essential  category  to  retain  sailors  in  the  fleet,  then  the  increased 
costs  for  this  item  should  remain  in  cost  estimates.  But  "trending"  increased 
overhead  rates,  without  understanding  that  unused  capacity  is  being  encouraged 
by  those  overhead  trend  lines,  should  be  made  unacceptable.  Costs  probably  can¬ 
not  be  reduced  in  the  future.  But  the  growth  lines  can  be  made  to  grow  in  a 
decelerating  path,  rather  than  an  exponentially  accelerating  one. 

There  needs  to  be  an  understanding  that  program  cost  growth  will 
occur — that  there  is  a  trade-off.  A  low  but  reasonable  estimate  for  a  project  will 
result  in  greater  program  cost  growth  than  will  occur  if  the  estimate  is  too  high. 
The  higher  estimate  may  save  embarrassment  to  the  program  manager  and  cost 
estimator,  but  excessively  high  estimates  may  also  lead  to  higher  end  costs  for  the 
program.  One  trades  off  total  program  cost  from  one  program  to  the  next  for 
program  cost  growth  from  the  initial  estimate  to  the  end  cost. 

One  key  to  avoiding  this  dilemma  lies  in  the  intelligent  use  of  management 
reserves.  We  can  suggest  that  reserves  must  be  programmed,  but  they  should  be 
retained  centrally  and  not  become  identified  with  a  specific  project.  The  reason  is 
obvious:  Once  identified  with  program  X,  the  project  manager  and  the  contrac¬ 
tors  cannot  help  but  to  plan  on  that  reserve  being  available,  and  designs  will 
evolve  accordingly.  On  the  other  hand,  if  reserves  are  kept  centrally,  then  the 
system  design  will  not  evolve  with  the  reserves  essentially  "built-in,"  yet  valid 
needs  can  still  be  filled  when  they  do  arise.  One  will  be  less  likely  to  spend  fund¬ 
ing  which  only  might  be  available,  than  to  spend  funds  which  are  available. 
Figure  4  provides  the  graphics  on  the  resulting  reduction  in  the  cost  growth  of 
Figure  3  when  management  reserves  exist.  There  will  still  be  overruns,  probably 
in  excess  of  reserves  (as  shown>,  but  the  net  result  would  be  a  system  that  costs 
less. 

The  cost  community  is  well-equipped,  intellectually,  to  understand  the 
various  aspects— analytical,  production,  contractual,  managerial— which  must 
be  integrated  to  control  costs.  They  understand  the  trends  which  must  be  ac¬ 
counted  for,  and  they  have  the  analytic  talent  to  sort  out  those  trends  that  are 
unavoidable  from  those  that  are  not.  Indeed,  hardly  anywhere  else  will  there  exist 
the  proper  mix  of  analytic  talent,  engineering  experience,  pricing  experience, 
project  experience,  and  contracting  knowledge.  What  is  needed  to  bring  these 
talents  to  bear  is  a  cost  policy  change  which  emphasizes  the  search  for  the  reasons 
of  cost  growth,  and  slows  the  urge  to  perpetuate  trends.  || 
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Multinational  Programs 

Major  Michael  J.  Rendine,  USAF 

The  increasing  expense  of  acquiring  military  systems,  coupled  with  un¬ 
favorable  economic  and  political  conditions,  has  caused  most  industrialized  na¬ 
tions  to  expect  coproduction  offset  as  a  condition  of  foreign  military  sales  (FMS). 
Consequently,  U.S.  Government  programs  with  potential  for  foreign  military 
sales  also  have  potential  as  multinational  coproduction  efforts.  The  size  of  the 
F-16  program,  in  terms  of  dollars  and  complexity,  proves  the  validity  of  this  con¬ 
cept  and  serves  as  a  model  for  the  future. 

Successfully  producing  aircraft  components  in  five  countries,  with  final 
assembly  in  three  countries,  is  recognized  as  a  significant  technical  achievement. 
The  financial  management  of  such  a  program,  however,  is  usually  viewed  as  a 
bookkeeping  nightmare  rather  than  as  a  challenging  opportunity  for  a  unique  and 
rewarding  experience.  People  not  familiar  with  the  F-16  program  focus  on  the 
cost-accounting  complexity,  imagining  rows  of  "bean  counters"  slaving  over  their 
calculators,  producing  reams  of  billing  information.  Actually,  there  are  surpris¬ 
ingly  few  people  within  the  systems  program  office  who  are  engaged  in  the  finan¬ 
cial  management  of  the  program.  And  the  center  of  their  activity  is  not  account¬ 
ability,  but  the  unique  econometric  issues  that  are  the  heart  of  the  multinational 
consortium. 

This  paper  will  address  the  critical  financial  challenges  of  the  F-16  multina¬ 
tional  program.  The  basic  economic  tools  necessary  to  deal  with  these  challenges 
will  be  considered,  as  will  the  interrelationships  established  to  bring  these  solu¬ 
tions  together  in  a  rational  application.  Finally,  I  will  discuss  the  problem  of 
"how  to  get  there"  on  any  multinational  program. 

The  Challenge 

The  F-16  memorandum  of  understanding  (MOU),  signed  at  the  Secretary  of 
Defense  level  by  the  United  States,  the  Netherlands,  Belgium,  Norway,  and 
Denmark,  stipulates  that  the  industries  of  the  five  nations  shall  be  insulated  from 
potential  losses  or  gains  caused  by  inflation  or  fluctuations  in  currency  exchange 
rates.  This  premise  is  essential  because  the  multinational  consortium  is  an  in¬ 
tergovernmental  organization  that  is  not  in  a  position  to  commit  industries  to 
money-losing  propositions.  The  stipulation  triggered  unique  currency  exchange 
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provisions  and  national/multinational  indexing  of  industry  costs.  The  existence 
of  inflation  indices,  however,  begs  the  question  of  determining  program  costs  in 
terms  of  the  time  value  of  money.  The  MOU  addresses  this  issue  by  stipulating 
that  the  European  participating  government  aircraft  shall  have  not-to-exceed 
target  cost  of  $6,091  million  as  measured  in  1975  base-year  dollars.  The  base- 
year-dollars  concept  was  applied  to  synchronize  the  congressional/parliamentary 
budget  basis  within  each  country.  The  use  of  1975  dollars  allowed  the  cost  of  the 
program  to  be  established  in  (at  that  time)  current-year  dollars,  while  affording 
each  country  the  budgetary  flexibility  to  address  then-year  dollars  in  their  own 
governmental  manner,  which  differs  among  the  five  nations.  The  upshot  is  th^it 
the  letters  of  offer  and  acceptance,  contractual  documents  between  the  U.S. 
Government  and  each  of  the  European  governments,  are  denominated  wholly  in 
1975  base-year  dollars.  In  addition,  for  economic  and  industrial  purposes,  each 
letter  of  agreement  contains  over  40  individually  priced  subcases,  which  is  two  to 
three  times  the  number  of  subcases  in  a  conventional  foreign  military  sales  program. 

On  the  U.S.  side,  both  DOD  and  the  Congress  recognized  the  potential  cost 
savings  of  producing  a  large  quantity  of  aircra%  (348  for  the  Europeans,  650  for 
the  U.S.  Air  Force)  as  well  as  the  potential  for  increased  cost  owing  to  multiple 
manufacturing/assembly  locations,  initial  investments,  learning  curves,  and  the 
cost  premiums  associated  with  European  work.  To  ensure  that  the  taxpayer 
would  not  pay  a  higher  cost  for  a  consortium  F-16  (998  aircraft  produced)  than 
for  an  all-domestic  program  (650  aircraft  only),  which  would  be  a  violation  of 
public  law,  an  impact  of  coproduction  report  is  required  with  each  annual  U.S. 
Air  Force  F-16  budget  submittal  to  Congress.  The  intent  of  this  report  is  to  com¬ 
pare  the  actual  program  against  a  giant  "what  if'— what  if  there  were  no 
coproduction  program,  and  the  U.S.  Air  Force  F-16's  were  built  domestically? 
The  ability  to  continue  to  project  the  cost  of  a  hypothetical  program  in  a  credible 
manner  rests  upon  the  competence  of  the  estimators  and  their  grasp  of  the 
econometric  principles  involved. 

Two  subcommittees  of  the  F-16  Steering  Committee  oversee  and  assist  the 
financial  manager  in  defining  and  developing  solutions  to  the  above  re¬ 
quirements.  They  are  the  Contractual  and  Financial  Subcommittee,  composed  of 
m.iitary  representatives  of  the  five  air  forces;  and  the  Subcommittee  on  Industrial 
Matters,  primarily  made  up  of  civilians  from  the  various  ministries  of  economics. 
In  addition,  24  cost-sharing/allocation  accords  are  in  force  multinationally. 

With  considerable  visibility,  and  motivated  by  fundamental  elements  of  the 
MOU  and  multigovemmental  requirements,  the  F-16  proceeded  to  deal  with  cur¬ 
rency  exchange  protection,  multinational  inflation,  base-year  dollar  tracking, 
and  impact  (cost/benefit)  of  coproduction.  These  challenges  may  be  new  to  the 
DOD  financial  manager,  but  the  solutions  merely  require  application  of  basic 
econometrics,  which  run  consistently  through  each  of  the  four  challenges. 
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Currency  Exchange  Protection 

The  F-16  contracts/subconlracts/purchase  orders  are  denominated  in  the 
domestic  currency  of  the  seller.  If  a  U.S.  prime  contractor  purchases  a  component 
from  a  Dutch  firm,  the  contract  is  written  in  Ho'land  florins.  Likewise,  if  the 
Dutch  firm  establishes  a  purchase  order  with  a  Belgian  firm  for  raw  material  from 
which  the  component  will  be  manufactured,  the  purchase  order  will  be  priced  in 
Belgian  francs.  Although  this  results  in  thousands  of  contractual  instruments 
denominated  in  each  currency,  stability  is  achieved  through  MOU-stipulated 
fixed  rates  of  currency  exchange.  Thus,  the  value  of  all  instruments  can  be 
summed  up  in  U.S.  dollars  by  applying  the  fixed  rates.  The  contractors  obtain  the 
currencies  to  pay  one  another  from  the  F-16  Currency  Clearing  House,  utilizing 
the  fixed  rates  of  exchange.  Because  the  contractors  obtain  currencies  at  a  fixed 
rate,  regardless  of  open-market  rates,  they  are  insulated  from  gains  or  losses 
resulting  from  currency  fluctuations.  The  five  governments,  however,  obtain  cur¬ 
rencies  at  the  market  rates  to  supply  the  currency  clearing  house.  Thus,  the 
governments  bear  the  currency  costs  or  reap  the  benefits,  just  as  any  government 
contends  with  the  cost  of  inflation  when  purchasing  defense  materials.  To  operate 
a  currency  clearing  house,  the  financial  manager  needs  more  than  currency  and  a 
good  financial  record  system.  Proper  operation  requires  knowledge  of  the  value 
(then-year)  expenditure  profile  of  each  instrument  so  sufficient  currency  reserves 
are  on  hand.  Knowing  "how  much  "  and  "when "  is  the  key  to  the  currency  clear¬ 
ing  house,  and  is  a  common  requirement  of  the  next  challenge. 

Multinational  Inflation 

Financial  managers  are  intimately  familiar  with  the  OSD  indices  used  to 
represent  a  cross-section  of  U.S.  inflation.  Application  of  general  indices  to  the 
F-16  program  was  unacceptable,  because  the  MOU  stipulates  that  each  industry 
must  be  insulated  from  abnormal  economic  escalation.  Thus,  individual  treat¬ 
ment  of  each  coproducer  is  required  with  tailored  indices.  Under  a  multinational 
accord,  each  country  provides  specific  labor  and  material  indices  (both  projec¬ 
tions  and  actuals  as  they  occur)  published  on  a  semiannual  basis.  To  properly 
apply  these  indices,  the  base-year  value  of  work  performed  by  each  industry  for 
each  half-year  interval  of  the  program  (1976-85)  must  be  known  in  terms  of  both 
labor  and  material  (domestic/non-domestic). 

Contractors  are  protected  from  the  effects  of  inflation  by  comparing  the  pro¬ 
jected  inflation  index  assumed  in  the  contract  to  the  actual  inflation  index  ex¬ 
perienced.  Significant  differences,  positive  or  negative,  are  corrected  by  an 
economic  price  adjustment  to  each  contract  on  a  semiannual  basis.  It  is  to  be 
hoped  that  the  index,  or  indices,  selected  is  a  representative  proxy  for  actual 
inflation. 
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The  total  summation  of  all  indices,  weighted  by  the  work  performed  by  each 
affected  industry  element,  results  in  the  F-16  multinational  fighter  program  index. 
The  index  is  updated  each  half  year  to  adjust  for  engineering  changes,  new  work 
(simulators,  trainers,  spares,  etc.),  as  well  as  the  revised  national  indices.  The 
revisions  accumulate  the  economic  price  adjustments  with  the  index,  providing 
then-year/base-year  integrity.  The  task  of  making  these  adjustments  has  been 
computerized,  with  a  savings  of  2  man-years  per  semiannual  update,  and  a  reduc¬ 
tion  in  processing  time  from  6  months  to  1  week.  The  principal  elements  are  “how 
much"  (work)  and  "when"  (semiannually). 

Base-Year  Dollar  Tracking 

The  harsh  realities  of  present-day  economics  demand  quick  payment  of  con¬ 
tractor  expenses,  with  the  time  value  of  money  fluctuating  recently  between  10 
and  20  percent.  Although  the  contractors  are  paid  in  then-year  dollars,  a 
methodology  was  required  to  establish  the  base-year-dollar  value  of  every 
expenditure.  Since  all  40  subcases  for  each  of  the  four  European  letters  of  agree¬ 
ment  are  priced  in  base-year  dollars,  the  then-year  value  is  a  function  of  how  fast 
the  base-year  dollars  are  expended,  and  the  prevailing  inflation  rate.  The  overall 
"system  requirements"  for  F-16  financial  management  involved  evaluating  all 
expenditures  (then-year  dollars),  including  economic  price  adjustments  against 
160  subcases  (4  countries  X  40  cases),  de-escalating  to  base-year,  and  applying  to 
160  respective  base-year-dollar  subcases.  An  automated  case  management  system 
computer  program  accomplishes  this  task  for  the  F-16.  Financial  managers  input 
actual  then-year  dollar  expenditures  into  each  then-year  case,  and  make  ad¬ 
justments  in  base-year  dollars  to  the  base-year  case  expenditure  profile.  Dif¬ 
ferences  in  estimated  vs.  actual  expenditures  (base-year  dollars)  are  "spread"  in 
future  quarters,  and  each  case  is  re-escalated  to  arrive  at  a  new  then-year-dollar 
value:  while  the  base-year-dollar  case  value  remains  constant.  Although  this  may 
appear  as  computer  magic,  it  is  an  application  of  "how  much"  and  "when."  If  the 
value  of  each  disbursement  is  known  over  time,  keeping  track  in  320  subcases 
(160  base-year,  160  then-year)  is  another  computer  problem.  The  principles  of 
"how  much"  and  "when"  remain  the  same. 

Impacts  of  Coproduction 

The  almost  overwhelming  task  of  deriving  the  costs/benefits  of  coproduction 
can  be  defined  simply.  So  far,  certain  data  bases  have  been  defined  to  solve 
previous  challenges.  These  data  bases  include  information  concerning  work  f)er- 
formed  under  every  contractual  instrument  for  each  industry,  broken  out  by 
labor  and  material  over  6-month  increments,  the  prevailing  inflation  rates  of  each 
industry,  and  expenditure  rates  over  time.  If  additional  information  is  added  to 
this  data  base,  new  capabilities  can  be  created. 
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The  new  capability  required  is  the  ability  to  econometrically  model  each 
domestic  and  European  industry.  The  information  required  includes,  but  is  not 
limited  to,  overhead  pools,  business  basis,  and  learning  curve. 

Given  this  information,  on  a  mathematical  model  basis,  all  European  effort 
can  be  removed  from  those  industries  and  re-priced  under  domestic  conditions.  In 
addition,  overall  domestic  effort  can  be  reduced  from  998  aircraft  to  the  650  air¬ 
craft,  U.S.  Air  Force-only  program.  With  this  information,  the  estimated  "what 
if"  price  can  be  developed.  Certain  other  non-model  data,  such  as  recoupments 
for  research  and  development,  must  also  be  added  to  complete  the  impact 
analysis.  Because  new  actual  data  (then-year)  is  available  every  year,  but  the 
target  estimates  are  base-year,  the  capability  to  translate  base-year/then-year 
must  always  be  present. 

The  impact  of  the  coproduction  task  can  be  overcome  by  utilizing  the 
previous  "how  much"  and  "when"  information,  plus  new  information  on  an 
industry-specific  basis.  Although  oversimplified,  and  stretching  the  "how  much" 
and  "when"  philosophy  to  its  limits,  this  new  information  fits  the  mold. 

Interrelationships  and  Rational  Applications 

Figure  1  is  an  attempt  to  schematically  define  the  interrelationship  of  these 
econometric  elements.  Following  the  figure,  the  governmental  labor  and  material 
indices  are  applied  against  the  work  distributions  throughout  the  program.  This 
results  in  the  F-16  multinational  fighter  program  index.  Reapplying  the  index  to 
any  new  work  will  define  the  overall  amount  of  currency  required  within  the  cur¬ 
rency  clearing  house  for  currency  exchange  (offset  being  a  base-year  value).  The 
then-year-dollar  expenditure  profiles  (work  distribution  over  time)  can  be  used  to 
break  down  the  anticipated  currency  clearing  house  requirements  by  quarter. 
Overall  actual  subcase  expenditure  (then-year  dollars  including  economic  price 
adjustment)  are  de-escalated  by  the  multinational  fighter  program  indices  to  pro¬ 
vide  base-year  dollar  equivalents  for  the  base-year  track  of  the  automated  case 
management  system.  The  distribution  of  work  by  industry,  plus  additional 
industry  econometric  data,  provides  the  basis  for  the  impact  of  coproduction 
model,  with  multinational  fighter  program  indices  used  to  normalize  base- 
year/then-year  actuals  and  projections. 

Although  not  totally  utilized  to  date  within  the  F-16  system  program  office, 
the  rationalized  elements  can  be  worked  together  in  an  integrated  system.  Within 
the  F-16  program  many  of  the  software  systems  and  data  bases  were  developed 
independently  because  system  requirements  were  defined  as  managers  worked  to 
solve  a  particular  problem.  The  evolutionary  nature  of  the  solutions  may  make  a 
totally  integrated  system  overly  complex  and  insensitive  to  change.  Therefore, 
integration  may  require  development  of  the  overall  system  specification  in 
advance;  however,  the  basic  philosophies  still  apply. 
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Table  I  is  a  generalized  data  table  that  attempts  to  rationalize  data-base  re¬ 
quirements  and  summarize  the  econometric  areas  a  multinational  program 
should  anticipate  entering.  If  this  taole  appears  simple,  perhaps  it  is  because  the 
concept  isn't  as  complicated  as  might  be  expected.  Many  of  the  data  elements  are 
used  during  prime  contractor  negotiations,  or  after  contract  award.  The  signifi¬ 
cant  difference  within  a  multinational  coproduction  program  is  that  these  data 
elements  must  be  expanded  to  the  subcontract /vendor  level. 

How  to  Get  There 

In  the  process  of  identifying  the  fundamental  econometr"  jls  necessary  for 
multinational  financial  management,  many  unique  computer-driven  accounting 
efforts  were  glossed  over  as  mere  mathematical  exercises.  In  reality,  these  com- 
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TABU  I 
Data  Table 


CHALLENGE 

CRITERIA 

HOW  MUCH 

WHEN 

Currency 

Per  Contractor 

Purchase  Order 
Value 

Quarterly 

Expenditure 

Profile 

Inflation 

Per  National 
Industry 

Semiannual 
Over  Time 

Base-Year  Dollars 

Per  Subcase 

Then-Year  Dollar 
Disbursements 

Quarterly 

Impact  of 
Coproduction 

Per  Component 

Unit  Cost 

Business  Base 
Overhead  Pool 
Learning  Curve 

Per  Option 
Semiannual 
Semiannual 
Log  Units 

puter  solutions  were  the  result  of  months  of  codifying,  system  requirement  defini¬ 
tion,  algorithm  development  and  writing,  loading,  and  debugging  of  programs. 
Clearly  this  is  too  much  to  ask  of  the  most  time-pressed  financial  managers, 
estimators,  and  budget  analysts.  The  key  to  "how  to  get  there"  is  the  simple  ap¬ 
plication  of  the  existing  science  of  operations  research  to  the  financial  office. 
Within  the  Program  Control  Directorate  of  the  F-16,  operations  research 
specialists  and  acquisition  management  officers  work  on  an  integrated  basis  with 
financial  managers.  Acquisition  managers  holding  master  of  business  administra¬ 
tion  degrees  (with  economic  concentration)  serve  as  a  bridge  between  conven¬ 
tional  financial  procedures  and  the  applied  science  of  operations  research.  F-16 
experience  has  shown  that  these  three  backgrounds  (and  dedicated  people)  can 
overcome  the  most  persistent  and  complex  problems,  and  minimize  manpower 
requirements.  Programs  are  constantly  under  revision  to  streamline,  add 
capability,  or  reduce  run  time.  A  large  majority  of  the  programs  are  written  in- 
house  during  the  development  process.  The  largest  and  most  complex  are  written 
in  final  form  or  from  scratch  by  commercial  programming  contractors. 

The  Air  Force  Academy  firstclassmen  summer  research  program  was  also  an 
extremely  effective  and  economical  way  to  extend  operations  research  and 
econometric  talents.  As  many  as  six  cadets  per  year  contributed  sophisticated  and 
rigorous  computer-aided  solutions  to  the  data  base.  The  large  majority  of  these 
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computer  programs  are  available  for  use  in  other  acquisition  projects.  The  prob¬ 
ability  of  direct  application  with  no  modification  is  small.  With  or  without 
existing  software,  however,  solutions  to  multinational  financial  management 
challenges  exist  through  the  mixing  of  disciplines  and  sciences. 

Summary 

The  emergence  of  multinational  programs  brings  with  it  an  amplification  of 
current  tasks  and  a  number  of  new  efforts.  The  amplified  tasks  relate  to  the  data 
base  normally  required  for  any  major  weapon  system  prime  contract.  Both  con¬ 
tractual  data  and  pre-award  negotiation  information  are  required  fully  on  all  sub¬ 
contracts  for  multinational  programs,  with  matching  government  data 
(inflation).  New  elements  such  as  currency  management  and  base-year-dollar 
management  will  be  challenging  but  will  not  fall  outside  the  realm  of  good 
business  practices.  The  key  element  is  the  blending  of  the  amplified  current  work 
and  new,  unique  efforts  together  in  a  rational  manner. 

The  cornerstone  of  the  F-16  in  this  regard  has  been  the  integration  of  opera¬ 
tions  research  personnel  and  business  management  personnel  (M.B.A.  with 
econometric  concentration)  within  the  conventional  program  control  environ¬ 
ment.  This  interdisciplinary  approach  has  been  responsible  for  highly  effective 
solutions  to  unique,  multidimensional  problems.  || 
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In  February  of  1970,  the  Genera!  Accounting  Office  (GAO)  reviewed  the 
acquisition  of  major  weapon  systems  and  noted  that  substantial  cost  growth  was 
occurring  on  many  ongoing  programs.  By  March  1971,  the  GAO  was  reporting 
that  the  Department  of  Defense  (DOD)  had  made  a  good  start  toward  identifying 
the  causes  of  this  cost  growth.  Among  the  causes  were  (listed  in  order  of  impor¬ 
tance)  estimating  errors,  engineering  changes,  economic  changes,  schedule 
changes,  net  quantity  changes,  and  support  changes.  A  GAO  report  of  theory 
and  practices  of  cost  estimating  for  major  acquisitions  showed  that  serious  prob¬ 
lems  existed  in  cost  estimating  and  suggested  basic  criteria  that  should  be  used  in 
preparing  estimates.  The  Department  of  Defense  (DOD)  concurred  with  the 
GAO  finding  and  stated  that  corrective  actions  had  been  taken,  i.e.,  the  establish¬ 
ment  of  Defense  Systems  Acquisition  Review  Council  (DSARC),  and  cost 
estimating  and  review  groups  at  service  headquarters  and  system  command 
levels.  This  paper  will  describe  the  role,  mission,  and  functions  of  the  Resource 
Analysis  Branch  (Op-96D)  as  a  participant  in  the  Navy  weapon  system  acquisi¬ 
tion  review  process  and  Cost  Analysis  Program. 

Cost  analysis  in  the  Navy  in  the  past  10  years  has  evolved  through  the  issue  of 
directives  and  instructions  into  the  cost  analysis  program,  which  defines  respon¬ 
sibilities  and  procedures  for  estimating  the  costs  of  acquisition  and  ownership  of 
Navy  weapon  and  support  systems,  and  for  reviewing  and  validating  such  cost 
estimates.  The  purpose  of  the  cost  analysis  program  is  to  provide  decision-makers 
the  best  program  cost  estimates  and  assessments  possible. 

The  impetus  for  parametric  cost  analysis  in  the  military  services  occurred  in 
December  1971,  when  Deputy  Secretary  of  Defense  David  Packard  sent  a 
memorandum  to  the  secretaries  of  the  military  departments  concerning  two 
historically  significant  cost  analysis  initiatives.  Mr.  Packard  requested  that  each 
military  department  perform  an  independent  parametric  cost  analysis  on  each 
major  weapon  system  at  key  decision  points,  and  make  the  analysis  available  for 
each  Defense  Systems  Acquisition  Review  Council  review.  The  second  initiative 
was  to  identify  the  steps  the  military  secretaries  were  taking  to  improve  the 
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capability  to  perform  independent  parametric  cost  analysis,  and  to  use  them  in 
acquisition  management. 

In  January  1972,  Secretary  of  Defense  Melvin  Laird  sent  a  memorandum  to 
the  secretaries  of  the  military  departments,  stating  that  "it  would  appear  that  each 
Service  Secretary  should  have  a  staff  component  capable  of  preparing  independ¬ 
ent  parametric  cost  estimates."  This  component  should  be  responsible  to  the  serv¬ 
ice  secretary  and  organizationally  separate  from  program  proponents.' 

For  several  years  prior  to  1971  a  modest  cost  review  and  analysis  staff  existed 
on  the  Department  of  the  Navy  Secretarial  Staff.  This  organization  was  officially 
established  in  January  1972,  and  was  called  the  Cost  Review  and  Analysis  Direc¬ 
torate.  It  reported  to  the  Assistant  Secretary  of  the  Navy  for  Financial  Manage¬ 
ment  (ASN[FM])  and  was  responsible  for  critical  review  of  cost  estimates, 
including  selected  major  weapon  systems  acquisitions.  The  Chief  of  Naval 
Operations,  recognizing  the  need  for  an  independent  cost-estimating  group  on  the 
OPNAV  staff,  issued  a  memorandum  in  June  1971.  The  memorandum  directed 
the  establishment  of  an  OPNAV  cost  estimating  and  review  group,  entitled  the 
Resource  Analysis  Branch  {Op-96D),  in  the  Systems  Analysis  Division  {Op-96), 
within  the  Navy  Program  Planning  Directorate  (Op-090).  In  October  1971,  the 
position  of  Chief  of  Naval  Operations  (CNO)  Advisor  for  Resource  Analysis  was 
established,  and  was  designated  the  head  of  the  Resource  Analysis  Branch  (Op-96D). 

The  Secretary  of  the  Navy  (SECNAV)  issued  SECNAVINST  7000.19  in 
October  1972,  establishing  the  Department  of  the  Navy  cost  analysis  program. 
The  purpose  of  this  instruction  was  to  promulgate  policy  on  cost  estimating 
throughout  the  Department  of  the  Navy,  and  to  assign  responsibilities  for 
estimating,  validating,  and  reviewing  the  cost-analysis  program.  The  cost- 
analysis  program  applies  to  all  organizations  within  and  supporting  the  Navy 
that  are  :  /ed  in  the  planning,  programming,  and  budgeting  process 

associated  with  weapon  systems,  forces,  and  support  activities.  One  of  the  major 
functions  of  the  Navy  cost  analysis  program  is  to  support  the  weapon  systems  ac¬ 
quisition  review  process  by  providing  weapon  system  life-cycle-cost  estimates 
and  assessments  at  each  milestone  review.  The  SECNAVINST  was  reissued  in 
March  1975  as  SECNAVINST  7000.19B. 

The  Chief  of  Naval  Operations  issued  OPNAVINST  7000.17,  "Cost 
Analysis,"  in  January  1973.  This  instruction  provided  policy  guidance  for 
implementation  of  the  Navy  cost-analysis  program,  and  established  a  three-level 
evaluation  process  for  Navy  cost  estimates.  In  the  acquisition  of  new  weapon 
systems  and  equipment,  cost  estimates  are  prepared  by  the  Naval  systems  com¬ 
mands  for  review  at  the  Chief  of  Naval  Material  (CNM)  level,  the  CNO  level, 
and  the  SECNAV  level.  The  instruction  was  reissued  in  September  1975  as 
OPNAVINST  7000.17A. 

In  May  1973,  SECNAVINST  5420.172  established  the  Department  of  the 
Navy  Systems  Acquisition  Review  Council  (DNSARC)  to  provide  a  formal 
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mechanism  for  the  Secretary  of  the  Navy  to  receive  the  counsel  of  his  principal 
advisors  before  making  decisions  concerning  initiation  or  continuation  of,  or 
substantial  change  to,  major  weapon  systems  acquisition  programs.  The 
DNSARC  also  provides  a  forum  for  review  of  major  weapon  system  acquisition 
presentations  to  be  made  to  the  Office  of  the  Secretary  of  Defense.  The 
SECNAVINST  was  reissued  in  May  1976  as  SECNAVINST  5420.172B. 

In  October  1975,  the  Cost  Review  and  Analysis  Directorate  merged  into  Op-96D, 
and  the  CNO  Advisor  for  Resource  Analysis  became  the  SECNAV/CNO  Ad¬ 
visor  for  Resource  Analysis.  This  change  made  the  position  "double-hatted," 
reporting  to  the  SECNAV  through  the  ASN(FM),  and  to  the  CNO  through  the 
Director,  Systems  Analysis  Division  (Op-96). 

The  position  of  CNM  Advisor  for  Cost  Analysis/Estimating  (NMAT-016) 
was  established  in  October  1977.  The  CNM  Advisor  acts  as  a  special  assistant  to 
the  CNM  in  matters  of  cost  analysis/estimating  and  associated  policy,  and 
represents  the  CNM  in  meetings  on  weapon  system  costs  and  cost-estimating 
functions  in  the  systems  commands,  OPNAV,  and  OSD. 

Role  of  Op-96D  in  the  Navy  Cost  Analysis  Program 

The  SECNAV/CNO  Advisor  for  Resource  Analysis  directs  a  staff  of  10  pro¬ 
fessional  civilian  and  military  cost  analysts  organized  into  four  sections:  aircraft, 
electronic/sensor,  missile,  and  ship.  Each  section  has  a  section  head  and  one  or 
two  cost  analysts.  An  organization  chart.  Figure  1,  shows  the  relationship  of  the 
Resource  Analysis  Branch  (Op-96D)  to  the  CNO  and  ASN(FM).  It  should  be 
noted  that  the  SECNAV/CNO  Advisor  is  also  the  Branch  Head  of  Op-96D. 

The  SECNAV/CNO  Advisor  is  the  senior  authority  on  cost  analysis  within 
the  Offices  of  the  Secretary  of  the  Navy  and  the  Chief  of  Naval  Operations.  The 
SECNAV/CNO  Advisor  and  his  staff  (Op-96D)  generate  and  provide  an 
independent  estimate  and  assessment  of  life-cycle  costs  of  major  weapon  system 
acquisitions  during  major  milestone  reviews  (DSARC  0,  1,  11,  III).  This  independ¬ 
ent  cost  estimate/assessment  is  usually  based  upon  parametric  techniques  that  are 
generally  different  from  the  techniques  used  by  project  managers.  "Independent 
cost  estimate/assessment"  means  the  cost  analysis  group  is  organizationally 
separate  from  program  proponents:  and  that  the  group  is  not  a  proponent  or  an 
opponent  of  the  weapon  system  acquisition.  The  SECNAV/CNO  Advisor  pro¬ 
vides  a  critical  review  and  analysis  of  cost,  schedules,  performance,  and  other 
pertinent  financial  management  aspects  of  major  Navy  programs  tor  the  CNO 
and  ASN(FM)  prior  to  the  DNSARC  proceedings.  The  objective  of  the  independ¬ 
ent  estimate/assessment  is  to  advise  decision-makers  on  the  reasonableness  of  the 
project  manager's /systems  command  life-cycle-cost  estimates. 

In  addition  to  generating  an  independent  life-cycle-cost  estimate/assessment 
for  each  weapon  system/support  system  in  the  acquisition  process,  the  Op-96D 
staff  performs  the  following  functions: 
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Resource  Analysis  Branch  (OP-96D) 


—Develops  weapon  systems  cost  models;  develops  and  validates  cost  estimating 
relationships  for  major  weapon  systems  and  maintains  their  currency;  and 
develops  methodology,  techniques,  and  criteria  to  be  used  in  analyzing  proposed 
weapons  and  evaluating  the  cost  of  alternative  courses  of  action. 

—Analyzes  cost  estimates  for  cost-effectiveness  analyses,  and  works  closely  with 
groups  that  perform  effectiveness  analyses  to  ensure  that  all  costs  attributable  to 
the  systems  (both  direct  and  indirect)  have  been  included  in  the  analyses. 
—Performs  independent  economic  evaluations  and  cost-sensitivity  analyses  on 
major  weapon  systems. 

— Ensures  that  all  costs  are  displayed  in  a  weapon  system  cost  estimate,  e.g.,  total 
life-cycle  cost,  at  the  OPNAV  level. 

—Reviews  program  documentation,  such  as  the  decision  coordinating  paper 
(DCP),  operational  requirement  (OR),  and  mission  element  need  statement 
(MENS),  to  ensure  that  resource  annexes  are  consistent  with  current  Navy 
program  estimates. 

— Recommends  to  the  directors  of  special  studies  the  cost  techniques  to  be  used, 
and  the  type  of  cost  analyses  to  be  performed  in  order  to  achieve  the  objectives  of 
the  study  effort.  Reviews  studies  and  analyses  from  the  cost  viewpoint  to  examine 
relevance  and  impact  of  assumptions  and  conclusions,  and  determines  the  ade¬ 
quacy  and  validity  of  analytical  methods  and  techniques  used. 
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—Determines,  with  the  assistance  of  the  Chief  of  Naval  Material  activities  and 
through  liaison  with  Navy  components,  OSD,  and  contractors,  the  sources  of 
cost  data;  defines  and  directs  plans  for  acquisition  and  interpretation  of  the  data; 
establishes  a  weapon-systems  data  base;  and  uses  them  to  independently  evaluate 
cost  estimates  forwarded  to  the  CNO  and  SECNAV. 

—Presents  the  independent  parametric  cost  estimates/analyses  to  the  OSD  Cost 
Analysis  Improvement  Group  (CAIG)  at  program  major  milestones  reviews. 

— Represents  the  Department  of  the  Navy  as  the  senior  representative  to  the 
CAIG,  and  coordinates  Navy  and  CAIG  actions. 

— Generates  life-cycle-cost  estimates  or  assessments  for  programs  delegated  to  a 
DNSARC  or  CEB  review  level. 

— Provides  policy  direction,  guidelines,  and  advice  to  Navy  activities  on  the 
development  of  cost-analysis  procedures. 

Role  of  the  Project  Manager  in  the  Program  Review  Cycle 

The  weapon  system  acquisition  review  process  is  not  a  one-time  event  for  a 
major  program.  The  project  manager  knows  that  the  program  will  cycle  through 
the  review  process  at  least  four  times;  that  is,  DSARC  0, 1,  II,  III,  going  from  con¬ 
cept  formulation  to  production  and  deployment. 

When  a  major  weapon  system  is  scheduled  for  a  DSARC  milestone  decision 
review,  a  series  of  preparatory  activities  are  initiated  within  the  Navy.  Various 
organizations  in  OSD,  OPNAV,  the  Naval  Material  Command  (NAVMAT),  and 
system  command  headquarters  coordinate  their  schedules  to  determine  an 
appropriate  DSARC  meeting  date  and  prepare  the  required  analyses,  reports,  and 
briefings.  There  are  many  players,  spectators,  and  decision-makers  involved  as  a 
major  program  proceeds  toward  a  DSARC  milestone  review.  The  relationship 
among  the  principal  participants  in  the  DSARC  process  is  shown  in  Figure  2. 

The  project  manager  (PM)  initiates  the  program  review  cycle  within  his 
system  command,  briefing  the  command's  management  and  seeking  approval  to 
proceed  to  the  next  decision  point— the  Chief  of  Naval  Material  and  the  Acquisi¬ 
tion  Review  Board  (ARB). 

The  PM  briefs  the  ARB  on  the  status  of  the  program  including;  technical  per¬ 
formance,  test  results,  schedules,  alternatives,  quantities,  funding,  and  potential 
technical  risks.  His  purpose  is  to  demonstrate  to  the  ARB  and  CNM  that  the  pro¬ 
gram  is  ready  to  proceed  into  the  next  acquisition  phase.  The  PM  must  address 
mission  need,  capability,  and  deficiency,  as  well  as  program  definition,  objec¬ 
tives,  progress,  technical  risks  and  solutions,  pertinent  options,  life-cycle  costs 
and  funding,  business  plans,  support  plans,  and  contractual  strategies,  as 
appropriate  to  the  state  of  the  program  (DSARC  0,  I,  II,  or  III). 

After  the  PM  receives  approval  of  the  proposed  program  from  the  Chief  of 
Naval  Material  (CNM),  he  briefs  the  CEB  on  the  status  of  the  program.  The  CEB 
may  direct  the  PM  to  change  the  content  of  the  briefing  and  accent  issues  relevant 
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Program  Cost  Review  Cycle 
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to  the  efficient  and  timely  acquisition  of  the  weapon  system.  The  PM,  having  in¬ 
corporated  into  his  briefing  the  changes  directed  by  the  CNM  and  CNO,  will 
brief  the  Department  of  the  Navy  Systems  Acquisition  Review  Council,  on  the 
status  of  the  program.  Approval  to  proceed  to  a  DSARC  milestone  decision 
review  must  be  obtained  from  DNSARC. 

Before  the  DSARC  review,  the  PM  will  present  to  the  OSD  Cost  Analysis  Im¬ 
provement  Group  each  program  alternative  under  consideration  and  explain  how 
the  cost  estimates  were  derived.  Concurrently,  the  Resource  Analysis  Branch 
(Op-96D)  presents  the  independent  life-cycle-cost  estimate  to  the  CAIG,  and  a 
comparison  of  the  PM's  cost  estimate  and  the  independent  cost  estimate.  The  data 
base,  methodology,  and  estimating  relationships  used  to  generate  the  cost 
estimates  are  discussed  with  the  CAIG  members. 
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The  PM,  during  the  program  review,  will  present  to  the  DSARC  principals  all 
the  issues,  problems,  alternatives,  and  funding  that  were  presented  to  the 
DNSARC.  The  PM  must  obtain  approval  from  the  DSARC  principals  to  proceed 
with  the  acquisition  of  the  weapons  system,  i.e.,  to  another  milestone. 

Role  of  Op-96D  in  the  Program  Review  Cycle 

The  Resource  Analysis  Branch  (Op-96D)  analysts  are  generating  an  independ¬ 
ent  life-cycle-cost  estimate  at  about  the  same  time  the  PM  is  generating  his  pro¬ 
gram  cost  estimate  and  preparing  to  brief  the  program  status  at  the  system  com¬ 
mand  level.  The  ground  rules  and  assumptions  used  by  the  PM  to  generate  his 
cost  estimate  must  also  be  used  by  the  Op-96D  cost  analysts.  The  description  of 
the  program  for  cost-estimating  purposes  must  be  the  same  for  all  cost  estimators; 
for  example,  performance  characteristics,  design  features,  quantities,  schedules, 
and  escalation/inflation  indices  should  be  identical  to  generate  comparable  cost 
estimates. 

The  cost-estimating  approach  and  methodology  to  generate  program  cost 
estimates  by  the  PM  and  Op-96D  analysts  are  often  different.  The  PM  may 
generate  his  cost  estimates  using  engineering  (bottom-up)  methods,  and  Op-96D 
analysts  generate  their  cost  estimates  using  statistical/parametric  (top-down) 
methods.  When  the  program  is  approaching  a  DSARC  II  or  III  decision,  Op-96D 
analysts  frequently  visit  the  prime  contractor  (and  sometimes  major  subcontrac¬ 
tors)  to  discuss  development/production  aspects  of  the  program,  obtain  actual 
cost  data  on  prior  phases,  and  discuss  the  contractor's  cost  projections.  All  of 
these  trips  are  coordinated  with  the  PM.  As  the  program  goes  through  the  review 
cycle,  the  PM's  staff  and  Op-96D  cost  analysts  carry  on  a  continuous  dialogue  on 
how  the  respective  (PM's  and  independent)  cost  estimates  were  derived,  that  is, 
data  base  used,  learning/experiences  curve  assumptions,  competition,  and 
anything  that  may  have  a  significant  impact  on  program  acquisition  cost. 

Before  the  CEB  program  review,  pre-briefs  by  the  project  manager  and/or 
Op-96  action  officers  take  place  within  the  Navy  Program  Planning  Directorate 
(Op-090),  to  acquaint  the  OPNAV  managers  with  the  details  of  the  program  and 
any  potentially  controversial  issues.  The  OPNAV  review  cycle  starts  with  a  brief¬ 
ing  to  the  Director,  Systems  Analysis  Division  (Op-96),  and  Director,  General 
Planning  and  Programming  Division  (Op-90),  by  the  Op-96  action  officer.  The 
Op-96  action  officer  function  is  to  provide  an  objective  analysis  and  recommen¬ 
dations  to  Op-96,  Op-90,  and  Op-090  concerning  need  for  the  system,  pros  and 
cons  of  the  program,  alternatives,  program  structure,  and  cost  and  manning  im¬ 
plications.  A  CEB  preview  brief  is  held  to  ensure  that  the  project  manager  is  ready 
to  proceed  to  the  scheduled  CEB. 

Several  days  before  the  CEB  preview,  C)p-96D  analysts  prepare  an  Op-96D 
assessment  which  is  incorporated  into  the  Op-96  assessment  for  the  CEB  preview 
chairperson  (Op-90).  The  Op-96D  assessment  addresses  all  financial  aspects 
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which  are  potential  issues  (e.g.,  funding  shortfalls,  pros  and  cons  of  different 
alternatives),  and  usually  gives  a  comparison  of  the  preliminary  Op-96D  cost 
estimate  and  the  PM  estimate.  The  Op-96D  assessment  is  provided  to  the  PM  and 
the  cost  comparison  is  incorporated  into  the  PM's  briefing. 

On  the  day  of  the  CEB  preview,  the  Op-96  action  officer  and  the  Op-96D 
analysts  pre-brief  Op-96  and  Op-90  before  attending  the  CEB  preview.  The  CEB 
preview  chairperson  frequently  directs  that  changes  be  made  to  the  briefing,  such 
as  addition  or  deletion  of  procurement  alternatives,  schedule  changes,  and  con¬ 
sideration  of  different  procurement  strategies.  Some  changes  may  require  revi¬ 
sions  to  the  PM  and  Op-96D  cost  estimates. 

Op-96D's  preparation  for  the  CEB  is  similar  to  that  for  the  CEB  preview.  The 
Op-96D  independent  cost  estimate  is  revised /refined  and  the  assessment  is  pro¬ 
vided  to  the  Chief  of  Naval  Operations,  and  to  the  PM  for  incorporation  into  his 
briefing.  On  the  day  of  the  CEB,  the  Op-96  action  officer  and  Op-96D  analysts 
pre-brief  Op-96,  Op-90,  Director  of  Program  Planning,  and  Op-090,  Director  of 
Navy  Program  Planning.  It  should  be  noted  that  the  PM  knows  what  the  in¬ 
dependent  cost  estimate  is  before  the  CEB  because  of  the  continuous  dialogue  be¬ 
tween  both  staffs.  The  PM  and  Op'96D  cost  estimates  are  presented  at  a  high  level 
of  aggregation  (RDT&E,  investment,  operating  and  support),  and  at  the  major 
subsystem  level  for  the  acquisition  phase  and  the  first-level  element  structure  for 
operating  and  support  phase.  Any  differences  in  cost  estimates  at  the  aggregate  or 
element  level  are  highlighted  for  management,  such  as  initial  spares  and  average 
unit  flyaway  cost. 

About  3  days  before  the  DNSARC,  Op-96D  (in  his  SECNAV  Advisor  role) 
briefs  the  ASN(FM)  on  the  technical,  programmatic,  and  financial  aspects  of  the 
program,  and  on  the  independent  cost  estimate  and  assessment.  The  ASN(FM)  is 
generally  interested  in  such  issues  as  shortfalls  in  program  funding  in  the  next  2-3 
fiscal  years,  cost  growth  since  start  of  the  program,  type  of  contract  award,  and 
reasons  for  significant  differences  in  PM  and  Op-96D  cost  estimates. 

Two  days  before  DNSARC,  Op-96D  analysts  prepare  an  assessment  to  the 
CNO,  PM,  and  the  DNSARC  principals,  incorporating  any  changes  necessitated 
by  CEB  directions.  The  PM's  and  Op-96D  program  cost  estimates  are  presented 
to  the  SECNAV  (DNSARC),  identifying  any  major  differences  in  the  cost 
estimates  for  various  elements  in  the  RDT&E,  investment,  and  operating  and  sup¬ 
port  phases  of  the  program.  The  project  manager  will  usually  present  the  pro¬ 
gram  cost  estimate  and  the  independent  cost  estimate,  at  the  aggregate  level,  to 
the  DNSARC  members.  Op-96D  analysts  usually  attend  the  DNSARC. 

Op-96D  is  required  to  present  the  independent  cost  analysis/estimate  to  the 
OSD  Cost  Analysis  Improvement  Group  15  days  before  the  DSARC  review  date. 
Until  October  1980,  Op-96D  presented  brief  technical  and  programmatic  descrip¬ 
tions  of  the  program,  the  independent  cost  estimating  methodology,  the 
independent  estimate,  and  the  Op-96D  assessment  of  the  PM’s  estimate.  The  PM 
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attended  the  CAIG  briefing,  primarily  to  answer  technical/programmatic  ques¬ 
tions  that  might  arise.  Then  in  October  1980,  the  CAIG  issued  new  guidelines  as 
to  what  must  be  addressed  at  the  CAIG  briefing.  Op-96D's  function  was  essen¬ 
tially  the  same  as  before  but,  for  the  first  time,  the  PM  was  required  to  brief  his 
cost  estimate  and  methodology.  Since  that  time,  Op-96D  has  coordinated  the 
division  of  briefing  topics  with  the  PM.  In  practice,  the  PM  has  usually  presented 
a  condensed  version  of  his  DNSAkC  briefing,  along  with  his  cost  estimate  and 
methodology.  The  Op-96D  presentation  is  the  same  as  before  except  that  minimal 
technical  descriptions  are  included. 

The  OSD  CAIG  acts  as  the  principal  advisory  body  of  the  DSARC  on  matters 
related  to  cost.  The  CAIG  provides  DSARC  members  with  a  review  and  evalua¬ 
tion  of  independent  and  project  manager  cost  estimates  prepared  by  the  services 
for  presentation  to  each  DSARC.  These  cost  reviews  consider  all  elements  of 
system  life-cycle  costs,  including  research  and  development,  investment,  and 
operating  and  support.  (See  DOD  Directive  5000.4,  OSD  Cost  Analysis  Improve¬ 
ment  Group  (CAIGI,  October  30,  1980.  for  a  list  of  responsibilities.) 

After  the  CAIG  briefing,  CAIG  analysts  conduct  a  detailed  review  of  Op-96D 
and  PM  cost  estimates  and  methodologies.  This  may  start  several  days  before  the 
CAIG  briefing.  The  CAIG  analysts  are  interested  primarily  in  the  data  base  and 
techniques  used  in  developing  the  methodologies,  and  the  ground  rules  and 
assumptions  used  in  the  application  of  these  methodologies.  The  CAIG  analysts 
then  prepare  an  assessment  of  the  PM  and  the  Op-96D  cost  estimates,  and  fre¬ 
quently  develop  a  CAIG  independent  estimate. 

The  CAIG  assessmenf/independent  estimate  is  incorporated  into  a  memoran¬ 
dum  to  the  DSARC  principals  from  the  CAIG  chairperson.  The  CAIG  chairper¬ 
son  presents  an  assessment /estimate  at  the  DSARC  meeting.  The  PM's  and  the 
Op-96D  cost  estimates  are  presented  to  the  DSARC  principals  by  the  project 
manager.  Op-96D  analysts  do  not  attend  the  DSARC. 

The  program  review  cycle  from  the  start  of  the  Acquisition  Review  Board 
review  and  Op-96  pre-brief  to  its  conclusion  with  the  DSARC  milestone  review  is 
shown  in  Figure  3.  There  are  10  briefing  points  in  the  Navy  program  review  cycle 
for  the  project  manager  and  the  Op-96D  analysts  to  consider,  and  it  takes 
approximately  60-90  days  to  cover  all  the  briefing  points. 

Elements  of  Program  Review  By  Op-96D 

When  Op-96D  cost  analysts  prepare  an  independent  cost  estimate  for  a  pro¬ 
gram,  they  are  obligated  to  review  the  total  program  the  project  manager 
presents  to  the  CNO/SECNAV  for  a  decision.  The  purpose  of  the  review  is  to 
confirm  that  the  program  conforms  to  all  applicable  Department  of  Defense 
directives  and  instructions,  design-fo-cost,  life-cycle-cost,  and  CAIG  guidelines. 
Quantities,  schedules,  funding  profiles,  and  appropriations  must  be  reasonable 
and  consistent  within  the  acquisition  and  operating  phases  of  the  program.  Infla- 
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tion/escalation  and  cost-estimating  methodology  mus*  be  correctly  applied  to 
estimate  total  program  cost. 

The  Op-96D  program  review  will  usually  cover,  but  is  not  limited  to,  the 
following  items  as  each  program  proceeds  through  the  acquisition  review  cycle: 
— The  inflation  indexes  and  outlay  rates  used  by  the  PM  in  generating  current- 
dollar  costs. 

— The  cost-estimating  methodology  used  by  the  PM  in  calculating  flyaway  unit 
cost  and  life-cycle  cost. 

—The  funding  profiles,  quantity  buys  (by  year,  totals)  outlined  for  the  program. 
—The  quantity  of  training  rounds,  initial  spares  requirements  for  the  program. 
— The  development  plan,  test  program,  and  production  schedule  for  feasibility 
and  conformity  to  DOD  regulations. 

— The  appropriations  included  in  the  program  cost  estimate. 

—Adherence  to  OPNAV  instructions  on  standards  of  presentation. 

— Consideration  as  required,  of  the  impact  of  comjjetilion  in  the  program  cost 
estimate,  and  estimate  of  savings  due  to  competition. 

—Consistency  of  the  program  cost  estimate  with  the  amounts  shown  in  the  FYDP 
program  page. 

—Comparable  and  consistent  analysis  of  alternative  concepts  for  a  program  in 
terms  of  cost,  schedule,  and  quantities  (all  resource  requirements). 

—The  business  base  of  potential  contractors,  along  with  the  cost/availability  of 
materials,  and  the  impact  of  labor  costs  and  future  labor  rates  on  a  program  cost 
estimate. 

Summary  of  Op-96D  Functions 

The  functions  of  the  Resource  Analysis  Branch  can  be  summarized  as  follows: 
What  We  Do 

—Prepare  independent  weapon  system  life-cycle-cost  estimates; 

—Assess  system  command  cost  estimates; 

—Develop  cost  analysis  methods; 

—Represent  the  Department  of  the  Navy  at  CAIG  briefings  and  coordinate  Navy 
and  CAIG  actions. 

How  We  Do  It 

— Collect  cost/performance/design  data  on  weapon  systems; 

— Develop  cost  estimating  relationships  for  systems  and  subsystems; 

— Develop  cost  methodology; 

— Develop  weapon  system  cost  models; 

— Computerize  cost  models; 

— Calculate  weapon  system  life-cycle-cost  estimates. 
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When  We  Do  It 

— Systems  in  acquisition  process  (CEB/DNSARC/DSARC); 

— Systems  in  special  review. 

For  Whom  We  Do  It 
-CNO  and  ASN(FM); 

— DNSARC  principals; 

-OSD  CAIG. 

Why  We  Do  It 

— To  provide  an  independent  (non-proponent)  weapon  system  cost 
estimate/assessment; 

— To  assist  managers  in  decision-making. 

In  summary,  the  role  of  SECNAV/CNO  Advisor  for  Resource  Analysis  and 
his  staff  (Op-96D)  is  to  be  a  consultant  to  the  project  manager.  Office  of  ;he  Chief 
ot  Naval  Operations,  and  the  Office  of  the  Secretary  of  the  Navy  on 
cost/resource  issues  as  the  weapon  system  proceeds  through  the  acquisition 
review  cycle.  || 
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1  he  multicountry  and  multiperiod  payments  problem  is  unique  to  the 
North  Atlantic  Treaty  Organization  (NATO)  E-3A  program.  It  may,  however, 
be  the  model  (or  future  cooperative  development  and  procurement  within 
NATO.  The  present  structure  for  ensuring  equitable  payment  for  costs  for  the 
NATO  E-3A  program  is  unfair  to  most  of  the  nations  participating.  Germany  has 
been  particularly  vocal  about  the  inherent  inequity  in  the  approach  adopted  by 
the  NATO  airborne  warning  and  control  system  (AWACS)  Program  Manage¬ 
ment  Agency’s  Legal  and  Contracts  Committee. 

The  Problem 

The  problem  is  quite  simple.  Taking  the  NATO  E-3A  program  as  a  case  in 
point,  we  see  that  initially  each  nation  agrees  to  fund  its  share  of  the  procurement 
costs,  using  a  fixed,  constant-dollar  share  as  its  base  for  budget  calculations. 
After  the  fixed  share  is  agreed  to,  in  this  case  42  percent  for  the  United  States,  30 
percent  for  Germany,  the  next  step  is  to  cost  out  the  program  in  base-year  curren¬ 
cy  (in  the  case  of  the  NATO  program,  the  base  year  is  1977  and  the  currency  is 
the  U.S.  dollar).  Of  course,  the  funding  is  not  needed  all  at  once,  so  there  is  a 
yearly  flow  of  resources  decided  upon  in  consultation  with  the  prime  contractor. 
For  the  E-3A  program,  the  amount  needed  varies  over  the  6  years  of  the  procure¬ 
ment  phase  of  the  program.  Each  state  must  also  agree  to  the  full  costs  in  then- 
year,  or  inflated,  dollars.  Thus,  there  is  an  open-ended  commitment  that  cannot 
be  measured  in  advance. 

The  most  straightforward  method  of  handling  the  joint-funding  problem  is  to 
ask  each  state  to  make  up  its  share  of  the  costs  in  the  year  they  are  incurred. 
Thus,  if  $200  million  were  owed  in  the  first  year,  then  the  United  States  would 
pay  $82  million,  the  Germans  $60  million,  and  so  on. 

There  are  13  nations  in  the  E-3A  program,  and  not  all  can  pay  the  full  amount 
owed  in  the  year  due.  Inflation  is  a  variable  whose  impact  is  unknown;  but,  more 
importantly,  most  nations  do  not  have  excess  budget  authority  available  to  pay 
E-3A  costs  until  later  in  the  program.  Some  nations  can  pay  their  shares  in  the 
final  2  years;  some  in  the  middle  3  years;  and  some,  like  the  United  States, 
whenever  needed.  There  is,  therefore,  a  mix  of  different  patterns  of  payment  that 
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must  be  rectified  with  the  need  to  mamtain  a  balance  of  shared  contributions  to 
the  cost  of  procurement. 

The  program  can  go  forward  so  long  as  we  can  devise  a  payments  package 
that  preserves  at  least  the  semblance  of  the  original  fixed-dollar  shares.  While  the 
United  States  may  pay  90  percent  in  the  first  year  and  50  percent  in  the  second 
year,  it  may  also  only  have  to  pay  20  percent  in  the  third  year  and  20  percent  in 
the  fourth  year,  and  pay  nothing  at  all  in  the  last  two  periods.  The  cumulative 
sum  of  the  total  program  expenditure  weights,  measured  in  constant  dollars, 
times  the  yearly  share  to  be  paid;  i.e.,  90  percent,  50  percent,  20  percent,  20  per¬ 
cent,  0  percent  and  0  percent,  divided  by  the  total  program  cost  in  constant 
dollars,  yields  the  original  program  share  itself. 

By  applying  constant-dollar  weights  to  the  current-dollar  expenses,  it  leaves  it 
to  a  foss-of-the-coin  situtation  to  decide  if  the  final  payment  arrangement  is 
equitable  for  all.  This  is  essentially  what  the  United  States  and  the  11  other  coun¬ 
tries  have  decided  to  do.  Germany,  however,  balks  at  this  "simplistic"  approach, 
and  with  good  reason — it  is  inherently  unfair,  and  there  is  a  far  better  way  to 
handle  the  problem.* 

Letting  It  Be  Decided  by  Chance 

The  toss-of-the-coin  method  is  the  simplest  to  implement.  It  means,  essential¬ 
ly,  that  if  Germany's  share  of  the  constant  dollar  costs  in  1983  is  70  percent,  then 
it  must  also  pay  70  percent  of  the  current-dollar  costs  incurred  in  that  year.  The 
unfairness  here  is  that  70  percent  of  the  current-dollar  costs  may  be  far  greater 
than  90  percent  of  the  costs  the  year  before,  even  assuming  the  constant-dollar 
costs  are  exactly  the  same  for  both  periods  (if  inflation  were  greater  in  1983  than 
in  1982,  this  could  be  true). 

How  can  rational  states  justify  this  inequity?  One  justification  is  that  there  is 
an  inherent  time  value  of  money  that  must  be  taken  into  consideration  when 
judging  the  fairness  of  a  payment  method.  By  putting  money  in  early,  the  United 
States  incurs  a  loss  in  procurement,  or  even  additional  financing  charges  (as  the 
U.S.  Federal  Government  deficit  must  be  financed).  This  loss  is  compensated  by 
the  gain  derived  from  paying  less,  in  terms  of  current  dollars,  later  in  the  pro¬ 
gram.  There  is,  however,  a  better  way  to  handle  the  problem;  it  is  essential  that 
the  alternative  method,  and  not  the  current  AW  ACS  payment  approach,  serve  as 
the  role  model  for  future  collaboration  programs. 

An  Alternative:  the  Funds-Transfer  Method 

A  more  rational  approach  to  the  problem  of  AWACS  cost  sharing,  one  that 
conforms  with  our  requirement  that  the  obligations  of  the  participants  be 


1.  The  German’s  recent  note,  'Cost  Sharing,  Funds  and  Exchange  Rate  Problems  in  Collaborative 
Projects '  (Annex  II  to  Addendum  to  AC/^4-WP/661,  is  a  case  in  point.  They  recognize  the  need  to 
create  a  more  rational  method  for  sharing  program  costs  on  such  multinational  programs. 
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measured  in  real-dollar  terms,  and  that  those  who  fund  the  early  stages  be  com¬ 
pensated  by  the  late  payers  for  the  time  value  of  money,  is  the  “funds-transfer" 
approach.  It  preserves  the  negotiated  program  shares  as  a  basic  working  principle 
while  allowing  the  schedule  of  individual  dollar  payments  to  change.  The  fixed- 
share  obligation  in  real  terms  remains  the  ultimate  focus  for  deciding  that  each 
state's  contribution  is  equitable.^ 

There  is  a  difference  between  cost  escalation  due  to  program  changes,  and  in¬ 
creases  due  to  inflation  alone.  The  bottom  line  is  that  the  invoices  will  have  to  be 
paid  whether  cost  increases  are  due  to  program  change  orders,  or  to  general  infla¬ 
tion.  The  burden  of  paying  these  unanticipated  costs  should  not  fall  dispropor¬ 
tionately  on  the  late  payers,  since  it  is  unclear  who  is  responsible  for  such  unan¬ 
ticipated  program  change  costs.  Further,  these  costs  are  not  easily  separated  from 
those  asso'  lated  with  overall  inflation. 

There  is  potential  for  serious  conflict  over  the  interpretation  of  the  meaning  of 
"1977  dollars"  as  stated  repeatedly  in  the  multilateral  memorandum  of  under¬ 
standing.  The  terminology  "1977  base  dollar"  assumes  that  this  is  a  measure  of 
value.  As  such,  we  should  be  able  to  apply  a  discount  rate  to  reflect  the  oppor¬ 
tunity  cost  to  the  government.  Opportunity  cost  is  not  related  to  the  inflation  rate 
on  the  E-3A  procurement  itself.  A  good  proxy  for  this  implied  cost  is  the  U.S. 
Government's  borrowing  costs  as  reflected  in  the  treasury  bill  rate.  A  case  can 
thus  be  made  for  applying  this  rate  for  other  participants  in  a  U.S.  dollar-based 
program,  such  as  AWACS,  because  the  U.S.  dollar  reserves  of  foreign  govern¬ 
ments  are  usually  invested  in  tiese  government  securities.  It  is,  therefore,  the 
logical  discount  rate  to  be  used  when  determining  1977-dollar  equivalents  for  in¬ 
flated  program  costs. 

Finally,  the  opportunity  cost  or  time  value  of  money  argument  applies  only  to 
dollars  that  the  early  paying  states  fund  in  excess  of  their  negotiated  program 
shares.  The  United  States  can  legitimately  insist  on  compiensation  only  for  the 
marginal  payments  made  over  its  agreed  share  of  the  constant-dollai  costs  (42.1 
percent).  When  the  staggered-payment  schedules  are  looked  at  in  this  way,  we 
can  readily  interpret  that  any  fundings  in  excess  of  the  42.1  percent  limit  are,  in 
reality,  loans  to  cover  underpayment  by  other  participants.  This  marginal  ap¬ 
proach  results  in  a  method  which,  when  the  actual  cash  outlays  are  discounted 
over  time,  yields  an  allocation  of  the  program  cost  burden  according  to  the  fixed 
shares  agreed  upon  in  the  original  agreement. 

How  the  Funds-Transfer  System  Works 

The  fixed  share  of  the  basic  program  schedule  in  1977  dollars  is  affirmed.  The 
staggered  fixed-dollar  payment  schedule  is  accepted  as  a  valid  approach.  Under 

2.  This  is  the  same  principle  as  buying  a  car  either  for  credit  or  for  cash.  Each  person  is  paying  the 
same  base  price  but,  depending  upon  the  term  and  rate  charged  for  the  loan,  the  final  cost  to  each  car 
buyer  differs. 


142  H  Concepts 


H 


the  funds-transfer  method,  each  country  pays  its  yearly  share  as  determined  by 
the  second,  staggered-payments  schedule.  For  example,  if  Germany  is  responsible 
for  60  percent  of  the  fixed-dollar  costs  in  1983,  it  must  pay  60  percent  of  the 
current-dollar  costs  in  that  year. 

The  difference  between  the  actual  amount  paid  in  a  given  year  and  a  country's 
negotiated  fixed  share  is  then  tracked  with  each  quarterly  payment.  Using  the 
above  example,  the  difference  between  Germany's  60  percent  funding  and  its 
constant-dollar  share  of  only  30  percent  is  viewed  as  a  loan  to  nations  paying  less 
than  their  fixed  share  in  that  year.  An  interest-like  surcharge  accrual  is  calculated 
as  a  bookkeeping  entry  to  the  credit  of  the  countries  funding  the  excess.  Con¬ 
versely,  a  charge  at  the  same  rate  is  applied  to  nations  paying  less  than  their  fixed 
shares  in  the  given  period.  The  actual  rate  used  should,  in  any  event,  equal  the 
discount  rate  agreed  upon  by  the  member  nations  as  appropriate  for  valuing  1977 
dollars. 

Cash  outlays  associated  with  the  surcharge  balances  are  not  made  until  the 
program  is  completed.  These  surcharges  are  compounded  on  the  outstanding 
balance,  including  earlier  accumulated  surcharge  calculations  and  credits.  Over¬ 
payments  are  netted  against  balances  as  they  occur. 

To  the  extent  that  a  country  cannot  make  its  full  current-dollar  payments  as 
scheduled,  it  may  borrow  additional  funds  from  other  participants. 

After  the  procurement  phase  is  completed,  a  final  accounting  of  the  principal 
and  surcharge  balances  is  made.  By  definition,  the  debit  balances  of  the  net  bor¬ 
rowers  equals  the  credit  balances  of  the  net  creditors  so  that  a  simultaneous 
transfer  of  funds  can,  in  effect,  net  these  bookkeeping  records  at  the  close  of  the 
program. 

A  Hypothetical  Example  Using  NATO  E-3A 

To  see  how  the  new  system  works,  a  hypothetii  ample  using  inflation 
rates  that  are  somewhat  above  the  current  rates  forecast  is  presented  (in  Table  1). 
United  States  Treasury  bill  rates  are  used  to  value  the  under-  and  overpayments 
by  member  states.  These  rates  are  consistent  with  the  underlying  rate  of  future 
U.S.  inflation. 

United  States  government  payments,  using  the  current  system,  are  estimated 
to  equal  $1.3  billion,  or  about  37  percent  of  the  program  cost.  Thus,  the  United 
States  achieves  savings  equal  to  the  difference  between  its  fixed,  42.1  percent 
share,  and  its  actual  current-dollar  contribution.  The  line  "variable  %  less  42.1  %" 
represents  the  amount  of  under-  or  overpayment  for  each  year,  based  on  the 
original  fixed-dollar  share  of  each  state,  in  this  case  the  original  baseline  fixed- 
dollar  share.  The  next  line,  "Surcharge  @  T-bill  rate,"  shows  the  net  interest 
earned  or  paid  out  on  the  accumulated  balances.  As  each  yearly  overpayment  ac¬ 
cumulates,  principal  plus  interest,  there  is  a  credit  available  in  the  U.S.  account 


NuMwrf  in  (Mrentheses  represent  underp^yients  or  0utsUn<}lfl9  t>«1«nces. 
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until  1985.  The  total  required  payback  is  described  in  the  last  column  in  the  series 
"End-of-Year  Balance."  In  the  U.S.  case,  its  credit  of  $44  million  in  1984  becomes 
a  debit  of  $98.6  million  in  1985,  as  the  last  U.S.  payment  is  far  below  the  program 
costs  paid  out  in  that  year  by  late  paying  states.  The  United  States,  in  effect,  owes 
this  amount  to  the  other  countries. 

This  method  is  fair  to  all  nations  and  should  be  acceptable  when  it  is  time  for 
late  payers  to  fund  their  agreed  contributions.  Italy,  a  late  payer,  originally  con¬ 
tracted  for  a  5.6  percent  share  of  the  program.  If  the  toss-of-the-coin  approach 
were  followed,  Italy  would  ultimately  pay  7.8  percent  of  total  program  costs.  By 
applying  the  funds-transfer  method,  Italy's  share  of  total  costs  would  decline  to 
just  6.9  percent. 

Similarly,  Germany  pays  just  1  percent  more  for  its  present  payment  pattern, 
rather  than  2.5  percent  more  as  would  be  required  if  the  U.S.  method  is  applied. 
The  United  States  still  saves  money  while  being  fully  compensated,  correctly  this 
time,  for  its  early  payments.  The  U.S.  share  now  becomes  39.4  percent  of  the 
total  estimated  program.  This  is  still  2.7  percent  less  than  the  United  States' 
original,  fixed-dollar  share,  of  42.1  percent. 

Conclusion 

The  funds-transfer  method  is  a  pragmatic  and  equitable  solution  to  the  prob¬ 
lem  posed  by  multilaterally  funded  defense  programs  when  payment  phasing  dif¬ 
fers  from  fixed  yearly  shares.  It  is  a  generalized  approach  and  can  therefore  serve 
as  a  precedent  for  future  cooperative  projects.  The  funds-transfer  method's  ad¬ 
vantages  are  that  it  is  equitable  and  readily  implemented,  it  is  indifferent  to  cost 
growth  due  to  inflation  or  program  chat^ges,  and  it  can  easily  accommodate 
stretched-out  payment  schedules  or  alterations  in  individual  state  contributions 
without  undue  burden  placed  on  the  early-paying  states. 

This  new  approach  should  be  acceptable  to  Germany  as  well  as  to  other  na¬ 
tions.  In  the  Tornado  project,  a  joint  German-British-Italian  effort  to  develop  a 
multirole  fighting  aircraft,  Germany  is  advocating  a  final,  monetary  compensa¬ 
tion  between  participants  to  settle  residual  differences  in  work-shares  vs. 
purchase-shares.  It  is  logical  that  Germany  may  accept  a  side  calculation  of  the 
opportunity  benefits  and  costs  accruing  to  the  early  vs.  the  late  payers  in  the 
NATO  AW  ACS  program,  so  long  as  the  final  netting  out  takes  place  after  all  ac¬ 
tual  expenditures  have  been  made.  || 


Selecting  Estimating 
Techniques  Using 
Historical  Simulation 

Charles  A.  Graver 

T'he  development  of  cost  estimates  is  a  critical  part  of  the  cost-benefit  com¬ 
parison  of  alternative  government  programs.  The  specific  cost  estimates  will 
often  be  the  deciding  factor  among  alternatives,  and  can  be  the  death  blow  to  an 
entire  program  if  none  of  the  alternatives  is  affordable. 

But  what  cost-estimating  techniques  should  be  used?  There  are  two  classes  of 
commonly  used  estimating  techniques  that  are  appropriate  for  the  early  phases  of 
the  development  life  cycle  when  system  performance  characteristics  are  being 
selected  and  detailed  hardware  component  specifications  have  not  been  made. 
These  are  (1)  costing  by  analogy,  in  which  the  costs  of  similar  systems  are  used 
for  the  estimate;  and  (2)  cost-estimating  relationships  (CERs),  which  are  equa¬ 
tions  relating  cost  to  system  performance  characteristics.  The  former  is  dep>endent 
on  the  availability  of  analogies,  and  the  estimates  derived  are  interpolations  from 
the  existing  data.  CERs,  on  the  other  hand,  can  be  used  for  interpolations  as  well 
as  extrapolation.  The  latter  is  most  often  the  cost  analyst's  problem. 

Within  the  class  of  the  cost  estimating  relationships,  there  are  still  a  number  of 
choices  to  be  made.  What  performance  characteristics  drive  the  cost  and  what  is 
the  functional  relationship? 

The  usual  selection  procedure  is  to  first  identify  the  physical  and  performance 
characteristics  (independent  variables)  that  are  likely  to  be  cost-driving  variables. 
Next,  data  on  cost  and  these  variables  arc  collected  for  previously  built  systems. 
Then  a  number  of  functional  forms  relating  cost  and  some  or  all  of  the  perform¬ 
ance  characteristics  are  tried.  Least-square  curve-fitting  techniques  are  the  usual 
fools,  and  these  are  applied  after  the  functional  form  has  been  transformed  into  a 
linear  equation.  Statistics  derived  from  the  regression  analyses  of  the  least- 
squares  fit  are  then  typically  used  to  select  the  most  significant  variables  and  best 
functional  form.  The  index  of  determination  (or  R2)  is  commonly  used,  with  the 
best  equation  having  the  R2  closest  to  1.  Computer  programs,  such  as  six  curve  or 
stepwise  multiple  regression,  have  been  devised  so  that  a  specified  range  of 
variables  and  functions  can  be  processed  quickly. 

Problems  in  Selecting  Estimating  Techniques 

A  number  of  theoretical  and  practical  problems  arise  from  this  method  of 
picking  the  estimating  technique.  They  are  often  not  understood,  and  can  result 
in  serious  mistakes.  They  will  be  familiar  problems  to  the  seasoned  analyst. 

The  first  problem  is  the  definition  of  accuracy.  Naturally,  error  minimization 
is  the  goal.  But  what  type  of  error?  Is  the  analyst  interested  in  minimiziirg  ab- 
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solute  errors  or  proportional  errors?  Least-squares  curve-fitting  techniques  do 
both,  depending  on  the  functional  form  used.  For  example,  the  least-square  fit  of 
linear  equations  minimizes  absolute  error.  On  the  other  hand,  equations  of  the 
form  y  =  ax^  usually  have  the  least-squares  fit  applied  after  taking  logarithms  of 
both  sides  of  the  equation.  This  is  equivalent  to  minimizing  the  proportional 
error.  Thus,  different  concerns  about  accuracy  are  used,  often  without  the 
analyst's  being  aware  of  them. 

The  second  problem  is  comparability  of  selection  criteria.  When  a  number  of 
functional  forms  are  possible,  which  is  best?  Table  1  illustrates  the  problem.  Here 
six  curves  are  being  evaluated.  The  criterion  used  for  the  comparison  is  the  index 
of  determination  (or  R2),  and  the  closer  to  1,  the  better  the  relationship.  Curve  6 
is  therefore  the  best.  However,  an  examination  of  the  residuals  quickly  shows 
that  there  are  serious  problems  with  this  estimating  relationship.  The  error  for  the 
large  values  of  X  is  so  bad  that  negative  costs  are  predicted. 

The  problem  is  that  the  indexes  of  determination  are  not  comparable.  This  is 
because  the  index  is  calculated  on  a  least-squares  fit.  But  the  fit  is  not  applied  until 
the  candidate  curve  has  been  transformed  into  a  linear  form.  For  example,  the 
linear  form  of  equation  6  (Table  I)  is 

1/Y^A  +  {B/X) 


The  fit  criterion  is  then 

1  =  1 

and  A  and  B  are  picked  to  minimize  this  quantity.  The  index  of  determination  is 
calculated  from  this  fit,  and  hence  applies  to  how  well  1/Y  is  predicted,  and  not 
to  the  quantity  of  interest  Y.  Hence,  the  values  of  the  indexes  are  not  comparable. 

This  brings  up  the  third  problem.  What  is  the  interpretation  of  the  usual 
statistics  which  accompany  regressions?  Clearly  the  R^s  are  not  comparable.  But 
what  does  the  standard  error  of  the  estimate  mean,  and  how  does  one  interpret 
the  other  statistics?  In  fact,  the  statistical  assumptions  of  the  models  are  rarely 
stated  explicitly,  and  the  analyst  rarely  checks  to  see  that  his  data  fit  the  model. 
Linear  regression  assumes  an  additive  error  term  that  is  normally  distributed 
around  the  true  linear  relationship  with  a  variance  which  is  statistically  indepen¬ 
dent  of  the  value  of  the  input  variables.  The  model  form  y  =  axt’  assumes  a 
multiplicative  error  term  with  a  log  normal  distribution  and  a  variance  that  in¬ 
creases  with  increasing  input  variable  values.  Without  checking  these  assump¬ 
tions,  the  interpretation  of  the  output  statistics  is  questionable.  Yet  F-tests  and 
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TABLE  I 

Model  Comparisons 


XMEAN:  7.S  Y  MEAN:  114.79 


NUMBER 

CURVE 

INDEX 

A 

B 

1 

Y  =  A-FB*X 

.870942 

1.57802 

15.0134 

2 

Y  =  A*EXP(B*X) 

.734369 

12.9491 

.238688 

3 

Y  =  A*X-I-B 

.943895 

5.5841 

1.46224 

4 

Y  =  A-KB/X) 

.644278 

164.108 

-214.977 

5 

Y  =  1/(A-i-B*X) 

.45971 

.093564 

-8.481988-3 

6 

Y  =  X/(A*X-FB) 

.982073 

-1.79869  $-2 

.206394 

For  which  curve  are  details  desired  (number)  ?  6 


COEFFICIENTS; 

EXPECTED  VALUE  95PCT  CONFIDENCE  LIMITS 

A:  -1.79869  $-2  -2.39852  $-2  -1.20886  $-2 

B:  .206394  .188814  ..t23974 
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T-tests  are  still  made,  and,  along  with  R^s,  are  used  to  justify  selection  of 
variables  and  functional  forms. 

The  problem  of  interpretation  has  been  compounded  by  the  use  of  stepwise 
regression  techniques  to  select  the  model.  Prior  to  the  introduction  of  the  stepwise 
regression  technique,  candidate  CERs  had  to  be  hypothesized,  with  the 
hypotheses  presum;  y  based  on  engineering  rationales  or  some  other  criteria. 
The  need  for  this  ,jecification  was  operationally  removed  when  the  stepwise 
multiple  regression  routine  became  available.  Only  the  candidate  variables  and 
their  allowable  transformations  had  to  be  specified.  However,  when  the  stepwise 
routine  was  applied,  the  resulting  CER,  while  fitting  the  data  well,  often  had  no 
physical  rationale.  The  applicability  of  the  result  then  became  questionable,  even 
with  good  fit.  For  example,  suppose  a  hundred  different  CERs,  differing  in  form 
or  variable  selection,  are  tried.  It  is  not  surprising  that  one  or  two  will  fit  well 
enough  to  be  judged  significant  at  the  0.05  significance  level.  This  follows  from 
the  fact  that  the  CER  hypothesis  is  not  picked  a  priori,  but  is  the  result  of  finding 
the  one  that  fits  the  data  best  from  a  hundred  different  candidate  CERs.  As  such, 
this  fit  could  easily  represent  one  of  the  five  times  out  of  100  that  such  a  fit 
theoretically  occurs  by  chance  {at  the  0.05  significance  level). 

This  discussion  throws  into  doubt  one  of  the  central  assumptions  of  least- 
squares  curve-fitting— that  which  fits  the  past  best  will  predict  the  future  best.  For 
it  is  this  criterion  that  the  stepwise  regression  procedure  uses  to  choose  CERs.  Fur¬ 
thermore,  when  functional  forms  are  different,  the  criterion  is  not  applied  in  a 
comparable  way  (Problem  2).  As  a  result,  a  fourth  problem  arises— /s  the  best-fit 
selection  criterion  best? 

The  fifth  problem  is  plagued  by  this  question.  How  does  the  analyst  state  the 
estimating  relationship's  capability  to  make  extrapolations?  Predicting  the  cost  of 
procurements  that  represent  extrapolations  from  the  data  base  is  the  problem  that 
the  cost  analyst  usually  faces.  It  seems  we  are  always  required  to  estimate  the  cost 
of  a  bigger  or  faster  plane,  or  one  that  is  better  in  some  combination  of 
characteristics  than  those  procured  in  the  past.  While  using  the  criterion  of  "that 
which  fits  best,  predicts  best"  should  work  reasonably  well  for  cost  predictions 
that  are  interpolations  on  the  characteristics  present  in  the  data  base,  the  criterion 
yields  little  information  concerning  cost  predictions  of  procurements  which  repre¬ 
sent  extrapolations  from  the  characteristics  in  the  data  base.  Theoretically,  one 
should  calc  date  prediction  intervals,  but  these  are  so  wide  for  most  extrapola¬ 
tions  that  they  become  meaningless.  Thus,  a  means  for  estimating  the  accuracy  of 
techniques  to  make  extrapolations  is  required. 

The  final  problem  is  how  to  compare  a  wider  class  of  estimating  techniques. 
The  discussion  to  this  point  has  dealt  with  techniques  which  use  the  entire  data 
base.  But  this  excludes  costing  by  analogy  and  other  methods  which  systematically 
use  a  subset  of  the  data.  For  example,  make  a  linear  relationship  using  all  data 
with  characteristics  no  more  than  20  percent  different  from  the  procurement  to  be 
estimated.  One  cannot  even  calculate  the  usual  statistics  for  these  methods,  let 
alone  compare  them  in  a  fair  manner. 
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Historical  Simulation  Scope 

Historical  simulation  was  developed  by  the  author  in  1969  to  help  the  analyst 
address  these  problems.’  It  provides  an  independent  view  of  the  estimating 
technique  selection  process,  because  the  focus  is  changed  from  fitting  the  data 
best  to  predicting  best.  As  a  result,  historical  simulation  is  a  complementary  tool 
which  can  be  used  along  with  the  usual  methods  to  select  better  estimating  techniques. 

The  analyst  is  still  required  to  define  his  accuracy  concerns  (absolute  or  pro¬ 
portional  errors).  However,  this  choice  is  made  explicit  and  does  not  depend  on 
the  model  form.  The  selection  criterion  is  then  applied  in  a  comparable  fashion 
across  all  candidate  estimating  techniques.  This  includes  costing  by  analogy  and 
other  methods  which  cannot  normally  be  compared  to  CERs. 

Historical  simulation  provides  summary  measures  of  the  candidate  tech¬ 
niques'  estimating  value,  which  is  easy  to  communicate  and  not  obscured  by 
questionable  statistical  jargon.  In  addition,  statistically  valid  measures  of  the 
historical  simulation  results  have  been  derived  and  can  be  used  when  the  usual 
linear  regression  assumptions  are  warranted. 

When  the  data  is  properly  organized,  historical  simulation  provides  real  in¬ 
sight  into  the  ability  of  estimating  techniques  to  make  extrapolations.  The 
measure  depends  on  how  well  these  extrapolations  would  have  been  made  in  the 
past.  Thus  it  gives  a  more  pragmatic  measure  than  the  usual  statistical  prediction 
intervals. 

Estimating  Procedures 

A  wider  perspective  of  the  estimating  process  is  needed  to  set  the  stage  for  the 
historical  simulation  development.  This  is  provided  by  viewing  the  problem  as 
selecting  a  cost-estimating  procedure  instead  of  just  a  CER.  The  distinction  be¬ 
tween  these  two  concepts  is  given  below. 

A  cost-estimating  procedure  consists  of  an  equation  form  (EF)  plus  a  tech¬ 
nique  for  estimating  the  values  of  the  parameters  (in  the  EF)  from  some  sample. 
An  example  of  an  estimating  procedure  is: 

Equation  Form 

Y  -  a  -t-  bX 
where 

Y  —  production  cost  of  the  item  to  be  estimated 

X  —  the  weight  of  the  item  to  be  estimated 

Technique 

Least-squares  curve  fit 

1.  C.  A.  Graver,  Historical  Simulation:  A  Procedure  for  the  Evaluation  of  Estimating  Tech¬ 
niques,  Volume  I,  "Procedure  Development  and  Description,"  and  Volume  II,  "Some  Examples," 
General  Research  Corporation  CR-0364-1,  June  1969. 
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II 


TABLE  II 

Cost  Estimating  Procedures 


PROCEDURE 

NUMBER 

EF 

TECHNIQUE 

1 

Y  =  a  +  bV 

Leait-squarei  fit 

2* 

Y  =  a  +  bV 

Line  determined  by  the  closest  two 
data  points  in  terms  of  V 

3* 

Y  =  a  bX 

Same  as  above,  except  closest 
measured  in  terms  of  X 

4 

Y  =  aXb 

Least-squares  fit  on  log  Y  =  log 
a  -t-  b  log  X 

where 


Y  K  production  cost 
X  =  weight 

V  =  volume 


*Procedures  2  and  3  are  very  close  to  costing  by  analogy.  In  effect,  the  analyst 
assumes  that  if  he  forms  a  line  with  the  two  closest  data  points  (in  terms  of  his 
independent  variable)  to  the  point  he  wishes  to  predict,  the  estimate  using  this 
line  will  be  better  than  an  estimate  made  using  a  line  that  fits  all  the  data. 


A  new  estimating  procedure  results  from  choosing  a  new  ER,  a  new  tech¬ 
nique,  or  both.  Hence,  the  combinations  given  in  Table  II  are  all  examples  of 
alternative  estimating  procedures. 

When  a  cost-estimating  procedure,  with  EF  Y  =  a  +  bX,  say,  is  used  in  con¬ 
junction  with  a  particular  sample  (i.e.,  a  particular  set  of  observations),  there  is 
derived  an  explicit  cost-estimating  relationship  (CER);  for  example 
Y  =  10  -h  25X.  This  is  a  result  of  estimating  the  EF  parameters  by  applying  the 
parameter  estimating  technique  to  the  given  sample.  Thus,  every  CER  has  iden¬ 
tified  with  it  a  particular  sample  and  an  estimating  procedure  consisting  of  an  EF 
and  a  technique.  The  relationship  of  these  entities  is  pictured  in  Figure  1. 

Historical  simulation  evaluates  an  estimating  procedure  by  varying  the  sample  to 
be  used.  This  is  described  in  the  next  section. 
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FICURI  1 

Relationship  of  CER  and  Cost  Estimating  Procedure 


Historical  Simulation  Description 

The  job  of  a  cost  analyst  is  to  try  to  predict  the  cost  (in  constant  dollars)  of  a 
proposed  future  procurement.  He  has  at  his  disposal  a  description  of  the  proposed 
procurement  in  the  form  of  a  set  of  physical  and  performance  characteristics.  In 
addition,  he  has  available  physical  and  performance  characteristics  and  cost  data 
in  constant  dollars,  for  example,  the  100th  unit,  on  similar  past  procurements. 
Hence,  his  primary  objective  is  the  prediction  of  a  future  procurement  using 
available  historical  data. 

Historical  simulation  uses  this  primary  objective  in  measuring  the  value  of  a 
cost-estimating  procedure.  This  basic  tenet  can  be  stated  as  follows;  The  cost¬ 
estimating  procedure  which  can  best  simulate  predictions  that  would  have  been 
made  in  the  past  will  actually  be  best  able  to  predict  the  future. 

To  evaluate  different  cost-estimating  procedures,  using  the  tenet  just  stated, 
historical  simulation  calls  for  each  candidate  cost-estimating  procedure  to  be 
tested  on  subsamples  of  the  actual  data  base.  For  each  subsample,  the  candidate 
cost-estimating  procedure  is  used  to  predict  the  cost  of  procurements  built  after 
any  of  the  procurements  in  the  subsample. ^  These  predictions  are  then  compared 
to  the  actual  costs. 

To  demonstrate  this  process,  consider  the  following  example  comprising  the 
13  data  points  listed  in  Table  III.^  The  data  have  been  ordered  as  to  date  of  first 


2.  This  ordering  of  the  data  base  oi< .  '•t  a  requirement.  If  the  estimate  to  be  made  requires 

an  extrapolation  of  a  particular  performance  .  ..acteristic,  then  the  data  should  be  ordered  on  the 
value  of  that  performance  characteristic. 

3.  N.  K.  Draper  and  H.  Smith,  Applied  Regression  Analysis  (New  York:  John  Wiley  h  Sons, 
Inc.),  1968. 
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TABLE  III 
Sample  Data 


PROCUREMENT 

NUMBER 

FIRST 

DELIVERY 

ACTUAL 

100th 

UNIT  COST 

Xl 

X2 

1 

1960 

95 

1,996 

153 

2 

1961 

31 

967 

144 

3 

1963 

60 

2,414 

149 

4 

1964 

82 

4,418 

144 

5 

1966 

25 

852 

107 

6 

1968 

67 

2,072 

136 

7 

1970 

243 

10,408 

177 

8 

1971 

54 

2,643 

160 

9 

1972 

112 

3,786 

172 

10 

1973 

106 

3,335 

203 

11 

1974 

183 

6,374 

196 

12 

1975 

156 

7,092 

187 

13 

1977 

177 

10,304 

167 

delivery  (second  column),  and  the  actual  cost  and  the  independent  variables  X] 
and  X2  have  been  collected  for  each  data  point:  Xi  and  Xz  are  physical  or 
performance  characteristics  (such  as  weight  and  speed)  which  should  be  useful  in 
specifying  the  cost  of  the  procurements  to  be  estimated.  The  following  cost¬ 
estimating  procedure  has  been  hypothesized: 

(1)  Cost  =  a -b  +  ^2X2 

where  a,  bj  ,  and  bz  are  to  be  estimated  through  the  process  of  a  least-squares 
curve. 

A  subsample  of  five  items  will  first  be  used;  that  is,  the  first  five  rows  of  Table 
III  are  treated  as  the  data  base.  This  is  the  data  base  from  which  an  analyst  would 
have  had  to  make  cost  predictions  in  1967.  Using  a  least-squares  fit,  the  derived 
CER  is 


(2)  Cost  =  -73.9  +  0.0104X2  +  0.792X2 

From  Table  III,  Xj  and  Xj  for  procurement  number  6  are  2,072  and  136.  If 
these  values  are  substituted  into  the  CER  of  equation  1,  the  predicted  cost  is  55.3. 
This  represents  a  prediction  because  the  sixth  procurement  is  not  in  the  subsample 
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TABLE  IV 

Predicted  Costs  Using  First  Five  Procurements 


PROCUREMENT 

NUMBER 

ACTUAL 

UNIT  COST 

PREDICTED 

COST 

RESIDUAL* 

6 

67 

55.3 

-11.7 

7 

243 

174.4 

-68.6 

8 

54 

80.2 

26.3 

9 

112 

101.6 

-10.4 

10 

106 

121.5 

15.5 

11 

183 

147.5 

-35.5 

12 

156 

147.9 

-8.1 

13 

177 

165.4 

-11.6 

*  Negative  numbers  are  underestimates;  positive  numbers  are  overestimates. 


data  base.  From  Table  III  the  actual  cost  v^as  67;  thus  the  cost  is  underestimated 
by  U.7. 

Next,  equation  1  can  be  used  to  predict  the  remaining  data  points  7-13.  These 
predictions  can  be  compared  to  the  actual  costs,  and  residuals  calculated,  yielding 
the  results  given  in  Table  IV.  As  one  can  see,  there  were  six  underestimates  and 
two  overestimates. 

The  entire  process  described  thus  far  is  now  repeated  for  a  subsample  size  of 
six.  That  is,  the  sixth  procurement  is  added  to  the  subsample,  taking  the  six  top 
rows  of  Table  III  as  the  data  base.  This  data  base  is  the  one  from  which  a  cost 
analyst  would  have  made  his  cost  prediction  in  1969.  Making  a  least-squares  fit  to 
this  data  base,  the  following  CER  is  obtained; 

(3)  Cost  =  -68.4  +  0.0105X;j  -1-  0.76X2 

When  comparing  equation  3  with  equation  2,  it  can  be  seen  that  the 
parameters  have  changed,  although  not  by  any  great  amount.  This  change  is,  of 
course,  the  result  of  adding  procurement  number  6  to  the  sample.  The  point  to  be 
remembered  is  that  the  explicit  CER  has  changed,  but  the  CER  form,  i.e.. 
Cost  =  a  biXi  +  biXj.,  and  the  parameter  estimating  technique,  namely, 
least  squares,  has  not  changed.  It  is  the  cost-estimating  procedure:  i.e.,  the  CER 
form  and  the  parameter  estimating  technique,  that  is  being  evaluated  by 
historical  simulation  and  not  any  one  explicit  CER  such  as  equation  3. 

Predictions  and  residual  calculations  for  procurements  7-13  can  now  be  made 
using  equation  3  yielding  the  results  shown  in  Table  V.  Notice  that  procurement 
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TABU  V 

Predicted  Costs  Using  First  Six  Procurements 


PROCUREMENT 

NUMBER 

ACTUAL 

UNIT  COST 

PREDICTED 

COST 

RESIDUAL 

7 

243 

176.1 

-66.9 

8 

54 

81.7 

27.7 

9 

112 

102.8 

-9.2 

10 

106 

121.8 

15.8 

11 

183 

148.3 

-34.7 

12 

156 

149.0 

-7.0 

13 

177 

167.4 

-9.6 

TABU  VI 
Predictions 


SAMPLE 

For  Sample  Point  Number 

SIZE  USED 

6 

7 

8 

9 

10 

11 

12 

13 

5 

55.3 

174.4 

80.2 

101  .€ 

■ 

147.9 

165.4 

6 

176.1 

81.7 

102.8 

K  K-B 

149.0 

167.4 

85.4 

114.3 

K 

177.4 

183.9 

227.0 

161.5 

172.6 

229.3 

110.7 

166.3 

176.1 

229.2 

164.5 

174.9 

229.8 

H 

179.7 

223.7 

227.1 

number  6  is  not  included  since  it  was  part  of  the  data  base  used  to  derive  equation  3. 

The  procedure  described  thus  far  can  be  repeated  using  subsample  data  base 
sizes  of  7,  8,  and  on  up  to  13.  In  the  last  case  the  entire  sample  is  used,  and  the 
usual  least-squares  fit  is  obtained.  Of  course,  no  predictions  for  which  an  actual 
cost  exists  in  the  data  base  can  be  made  using  the  final  CER.  However,  this  is  the 
CER  which  will  be  used  to  make  the  future  prediction,  if  the  estimating  procedure 
being  evaluated  by  historical  simulation  is  chosen  as  the  best  method  for  predict¬ 
ing  cost. 
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TABLE  VII 
Residuals 


The  outputs  described  can  be  conveniently  summarized  in  a  table  of  predic¬ 
tions  (Table  VI),  a  table  of  residuals  (Table  VII),  and  a  table  of  parameter 
estimates  (Table  VIII).  The  interpretation  of  this  output  will  be  discussed  later. 

Generalization  of  the  Procedure 

In  the  reference  cited  in  footnote  1,  the  above  procedure  is  generalized,  in 
terms  of  mathematical  notation,  which  can  then  be  applied  to  a  wide  range  of 
estimating  procedures,  including  analogy.  Thus,  the  same  evaluation  procedure 
can  be  used  to  compare  a  wide  class  of  estimating  procedures. 


Concepts 


Information  on  Extrapolation 

In  the  example,  the  data  were  ordered  in  the  time  dimension;  however,  this 
need  not  always  be  the  case.  For  example,  the  data  set  in  Figure  2  has  been 
ordered  in  the  X-dimension,  and  the  prediction  problem  represents  a  further  ex¬ 
trapolation  in  the  X-direction. 

The  candidate  estimating  procedure  is 

Cost  “  a  +  bX 

and  a  least-squares  curve-fitting  technique  is  used  to  pick  the  parameters. 

.  At  the  first  stage  of  historical  simulation,  the  first  three  data  points  {Pi,  Pi, 
and  P3)  are  used  to  fit  a  line  J  |.  The  estimate  of  P4  would  be  low  by  the  amount 
Ri.  At  the  next  stage  of  historical  simulation,  line  fi  would  be  derived  using  as 
the  data  base  points  Pj,  Pi,  Pj,  and  P4.  The  estimate  of  P5  derived  from  {3  would 
be  low  by  Ri.  The  process  is  continued  deriving  lines  (3  from  data  points  Pj 
through  Pg,  and  (4  from  Pi  through  P4.  The  estimates  of  P*  (from  {3)  and  P7 
(from  {4)  are  low  by  R3  and  R4,  respectively.  Thus,  all  the  predictions  obtained 
were  low. 


FIGURE  2 

Theoretical  Ex.  nple 
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Looking  at  each  of  the  iines,  however,  it  does  not  seem  that  the  fit  (to  the  data 
they  were  derived  from)  is  too  bad.  In  fact,  (I4  would  probably  be  accepted  as  a 
good  model  for  the  first  six  data  points,  using  statistics  based  on  regression 
theory. <  Hence,  the  model  would  be  accepted  using  the  regression  theory 
statistics,  while  it  would  be  rejected  using  historical  simulation. 

Of  course,  in  this  case,  a  simple  plot  of  the  data  would  convince  an  analyst 
that  he  has  the  wrong  model  (it  should  be  exponential  rather  than  linear).  This, 
however,  is  a  consequence  of  a  two-dimensional  problem  (cost  and  X)  in  which 
plots  can  be  made  and  our  illustration  could  be  drawn.  The  analyst  will  not  have 
the  luxury  of  such  plots  when  working  with  more  than  one  independent  variable, 
and  an  extension  of  this  example  to  a  multiple  independent  variable  model  can 
readily  be  made  (without  a  figure,  however). 

Output  Interpretation 

In  interpreting  the  results  of  historical  simulation  (or  indeed,  to  make  in¬ 
ferences  from  the  usual  regression  statistics),  the  analyst  is  examining  two  basic 
questions  about  the  cost-estimating  procedure  under  study: 

—Is  the  estimating  procedure  valid;  i.e.,  is  it  a  true  representation  of  the  cost 
generating  process  under  study? 

—How  reliable  is  the  estimating  procedure;  i.e.,  is  the  model  variance,  and  hence 
the  variance  in  the  estimates,  large  or  small? 

The  answers  to  these  questions  are  used  by  the  analyst  to  choose  between  dif¬ 
ferent  candidate  cost-estimating  procedures  (ranking),  to  define  new  candidate 
cost-estimating  procedures,  and  to  make  estimates  about  the  prediction  accuracy. 
Ways  of  using  the  historical  simulation  output  for  these  purposes  are  discussed 
below. 

Direct  Examination  of  Historical  Simulation  Output 

A  direct  examination  of  the  contents  of  the  output.  Tables  VI,  VII,  and  VIII, 
can  add  insight  into  the  question  of  model  validity,  the  identification  of  ques¬ 
tionable  sample  points,  and  the  identification  of  new  candidate  estimating  pro¬ 
cedures.  For  example,  the  following  inspections  can  be  made: 

1.  Each  column  of  Table  VII  gives  the  residuals  for  a  particular  sample  point. 
One  can  ask  if  these  residuals  are  improving  (getting  smaller  in  an  absolute  sense) 
as  the  sample  size  grows  (that  is,  as  the  analyst  looks  down  the  column).  One 
would  expect  the  residuals  to  improve— or  at  least  not  get  any  worse— if  the 


4.  It  should  be  noted  that  there  is  another  technique,  called  "time  sequence  plot  of  the  residuals." 
which  for  the  example  being  discussed  would  result  in  a  sequencing  of  residuals  from  the  usual  regres¬ 
sion  analysis  that  would  indicate  a  lack  of  fit.  However,  the  consequences  of  retaining  the  model  (in 
this  example),  i.e.,  the  likelihood  of  underestimates,  are  more  apparent  when  processed  by  historical 
simulation.  Furthermore,  even  though  residual  plots  should  be  analyzed  whenever  a  least-squares- 
curve  fit  is  made,  the  fact  is  that  such  examinations  of  residuals  are  often  forgotten. 
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model  is  valid  and  the  sample  consistent.  This  behavior  is  not  true  (or  the  test  run 
sample.  The  residuals  are  erratic  or  tend  to  get  worse  for  sample  points  9,  11,  12, 
and  13. 

2.  Are  there  any  consistent  errors?  For  example,  does  the  estimating  pro¬ 
cedure  consistently  underestimate  (have  negative  residuals)  most  sample  points? 
If  so,  then  the  cost-estimating  procedure  shows  signs  of  bias.  Again,  by  examin¬ 
ing  any  column  of  Table  VII,  one  might  find  sample  points  that  are  consistently 
under  or  overestimated  by  a  substantial  amount.  In  this  case  there  is  reason  to 
suspect  that  the  data  point  in  question  does  not  belong  to  the  population,  or  that 
errors  have  been  ihade  in  recording  its  cost  or  the  values  of  the  independent 
variables.  For  the  test  run  data  there  appears  to  be  no  indication  of  bias,  as  the 
residuals  are  neither  mostly  negative  or  mostly  positive.  There  are  sample  points, 
however,  that  show  substantial  consistent  errors,  such  as  points  7  and  13. 

3.  Residuals  along  any  row  of  Table  VII  are  all  derived  from  the  same  sub¬ 
sample.  Comparing  two  adjacent  rows  indicates  the  impact  on  the  prediction 
process  of  the  point  added  to  the  larger  subsample.  One  might  therefore  ask  if 
there  have  been  significant  changes,  in  some  consistent  manner,  from  one  row  to 
the  next.  If  so,  the  sample  point  added  is  dominating  the  estimating  procedure, 
and  if  the  changes  in  residuals  are  not  for  the  better  (i.e.,  smaller  absolute 
residuals),  then  the  question  of  whether  or  not  the  sample  point  properly  belongs 
to  the  population  is  again  raised.  As  an  example,  if  rows  for  subsample  sizes  of  six 
and  seven  data  points  are  compared  in  Table  VII,  we  see  substantial  changes  in 
the  residuals.  While  some  residuals  have  improved  (sample  points  9  and  11), 
others  have  definitely  become  worse  (sample  points  12  and  13).  There  is  no  ques¬ 
tion  that  sample  point  7  has  had  a  significant  impact,  but  its  impact  is  mixed. 

4.  Finally,  the  estimates  of  the  parameters  (Table  VllI)  can  be  examined.  Are 
they  reasonably  stable,  showing  signs  of  convergence  as  the  sample  size  grows?  If 
so,  then  one  feels  a  greater  assurance  of  the  model's  validity;  the  information 
concerning  the  values  of  the  model  parameters  is  essentially  the  same  from  all  the 
sample  points.  If  not,  then  there  might  be  something  in  the  pattern  of  the 
estimated  coefficients  that  would  suggest  a  new  candidate  cost-estimating  pro¬ 
cedure  or  that  would  identify  a  questionable  sample  point.  In  Table  VIII  it  can  be 
seen  that  the  desired  stability  did  not  take  place  for  the  test  run  data.  The  inclu¬ 
sion  of  sample  points  7  and  8  had  a  significant  impact  on  the  parameter  estimates. 
Hence,  these  points  ought  to  be  examined  carefully. 

In  summary,  there  is  a  great  deal  of  "look-see"  evidence  concerning  the  model 
validity  in  the  output  of  historical  simulation.  This  output  can  be  used  to  build 
confidence  in  model  validity,  or,  conversely,  aid  in  hypothesizing  a  new  cost¬ 
estimating  procedure.  In  addition,  it  can  help  to  identify  questionable  sample 
points.  Furthermore,  no  information  concerning  the  process  has  been  lost. 
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Data  Summarizations  That  Do  Not  Depend  on  Statistical  Assumptions 

Data  summarizations  (or  statistics)  discussed  here  have  the  property  that  they 
can  be  calculated  and  consistently  interpreted  for  any  candidate  cost-estimating 
procedure.  These  summarizations  can  thus  be  used  to  compare  all  candidate 
estimating  procedures. 

This  lack  of  dependence  on  statistical  assumptions  introduces  uncertainty  as 
to  what  data  summarization  is  best.  Hence,  several  different  summarizations  are 
suggested.  Arguments  for  their  use  are  necessarily  heuristic  in  nature,  and  the 
choice  of  which  particular  summarization  to  use  depends  on  the  analyst's 
accuracy  definition.  He  can  exercise  this  choice  by  picking  loss  functions  and 
weighting  schemes  best  suited  to  his  application. 

Before  describing  the  summarizations,  it  will  be  useful  to  identify  the  portion 
of  the  historical  simulation  output  that  will  be  used.  Only  the  values  from  the 
residual  table  (VIII)  are  used,  as  it  is  the  errors  of  prediction  that  are  of  interest. 
Which  of  these  residuals  to  use  is  not  entirely  clear. 

Using  all  of  the  residuals  is  appealing  in  that  no  information  will  be  thrown 
away.  However,  there  are  problems  involved  in  knowing  how  to  use  all  of  them 
fairly.  The  residuals  are  certainly  not  independent;  hence,  use  of  all  the  residuals 
introduces  problems  of  statistical  interpretation  and  weighting. 

If,  however,  only  one  residual  is  used  for  each  sample  point,  in  particular  the 
one  made  from  the  largest  available  subsample  size— the  entry  in  the  diagonal  of 
Table  VII — then  the  problems  of  weighting  and  statistical  interpretation  are 
greatly  reduced.  Furthermore,  this  selection  is  not  without  heuristic  justification. 
In  effect,  we  are  looking  at  the  prediction  made  from  the  largest  available  sub¬ 
sample  size  for  each  procurement.  These  are  the  subsamples  that  would  have 
been  used  and  predictions  that  would  have  been  made  if  the  cost-estimating  pro¬ 
cedure  had  been  used  in  the  past. 

For  notational  convenience,  let  us  relabel  these  residuals  by 

^"o  +  l . ^n' - % 


where  ng  was  the  minimum  sample  size  used  in  the  historical  simulation  and  N  is 
the  size  of  the  entire  data  base.  The  collection  of  these  residuals  will  be  referred  to 
as  R. 

The  question  being  addressed  in  this  section,  then,  is  how  to  summarize  the 
data  in  R  so  that  one  can  choose  between  several  estimating  procedures.  In  addi¬ 
tion,  it  will  be  useful  if  these  summarizations  indicate  how  well  the  estimating 
procedure  will  do  in  the  future. 


—  -.■K- 
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One  such  summarization  is  that  of  average  proportional  error.  It  is  calculated 
as  follows; 


(4) 


Average  Proportional  Error 


N 


E 


i=«o+i 


where 

yi  *  actual  cost  of  the  procurement  indexed  by  i 
no  =  minimum  sample  size  used  in  the  historical  simulation 
N  =  the  size  of  the  data  base 

The  average  proportional  error  should  be  used  when  one  is  worried  about 
proportional  rather  than  absolute  cost  errors.  In  addition,  this  measure  is  prob¬ 
ably  the  easiest  to  communicate.  Every  cost  analyst  has  been  asked  to  indicate 
how  reliable  his  prediction  is;  for  example,  is  it  within  ±20  percent?  Having 
calculated  the  average  proportional  error,  he  can  answer  this  query  by  saying, 
"The  cost  estimating  procedure  from  which  this  estimate  has  been  derived  has  an 
average  proportional  error  of,  say,  IS  percent,  which  implies  that  if  it  had  been 
used  to  make  these  types  of  predictions  in  the  past,  it  would  have  been  off  on  the 
average  of  15  percent."  Hence,  a  reasonable  answer  to  the  query  would  be  that  an 
error  of  ±  15  percent  should  be  expected. 

Contrast  the  above  answer  to  one  made  from  the  usual  regression  theory  out¬ 
put  utilizing  statements  of  F-tests,  T-tests,  R2,  prediction  intervals,  etc.  How 
aware  of  the  underlying  statistical  assumptions  or  the  meaning  of  these  statistics 
is  the  recipient  of  the  prediction  results?  Their  meaning  is  certainly  not  as  easy  to 
understand  as  the  average  proportional  error. 

One  drawback  to  average  proportional  error  is  that  it  places  the  same  em¬ 
phasis  on  predictions  made  from  a  sample  of  size  5  as  predictions  made  from  a 
sample  of  size  12.  For  any  cost-estimating  procedure  that  makes  use  of  every  data 
point  in  its  subsample,  this  equality  of  weighting  may  seem  unjustified.  After  all, 
predictions  should  be  getting  better  as  the  sample  size  increases.  Hence,  the 
following  weighted  average  proportional  error  is  suggested: 


N 

(5)  Weighted  Average  Proportional  Error  =  ^ 

i  =  no  +  l 


Vi 


The  weights,  of  course,  add  up  to  one 

E  Wi=l 
i  =  no  +  l 


i 
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and  vary  proportionally  with  the  sample  size.  They  can  be  as  extreme  as  assign¬ 
ing  all  weight  to  N,  which  is  a  choice  that  might  be  made  by  an  analyst  who  feels 
that  most  information  is  contained  in  the  one  prediction  made  from  the  largest 
subsample  size.  The  author's  preference  for  a  weighting  scheme  is 

N 

(6)  Wi  =  Si  /  E  S, 

i  =  no  +  l 

where  S,-  is  the  subsample  size  used  for  the  particular  prediction.  This  equation 
would  give  the  predictions  from  subsample  size  10  twice  as  much  weight  as  the 
predictions  from  subsample  size  S,  and  thus  is  in  accordance  with  the  notion  that 
if  the  estimating  procedure  is  valid,  then  predictions  should  improve  as  the  sam¬ 
ple  size  gets  larger.  Furthermore,  the  use  of  this  type  of  weighting  scheme  does 
not  effectively  change  the  simple  Interpretation  of  the  summary  statistic  discussed 
in  equation  4. 

Another  alternative  to  average  proportional  error  is  that  of  squared  average 
proportional  error,  i.e.: 

1  ^ 

(7)  Squared  Average  Proportional  Error  = -ry — —  ^ 

^  '  o  i  =  tto  +  l 

One  would  use  this  type  of  summarization  when  he  wishes  to  penalize  propor¬ 
tional  errors  in  an  exponential  fashion. 

Finally,  one  might  be  more  concerned  with  absolute  rather  than  relative  error. 
A  calculation  such  as 

1  ^ 

(8)  Average  Squared  Error  =s=  -  Yy  (Ri)^ 

^  l  =  no  +  l 

could  be  made.  Although  this  statistic  appears  to  be  similar  to  the  calculation  of 
the  variance  estimate  in  regression  theory,  the  residuals  in  question  here  are  based 
on  predictions,  not  fits. 

A  general  framework  for  these  summary  statistics  has  been  developed  in  the 
references  in  note  1.  Not  only  are  different  weighting  schemes  allowed,  but  one 
can  also  define  loss  functions  tailored  to  the  required  estimate.  For  example, 
overestimates  may  not  be  a  concern,  and  the  average  underestimate  may  be  the 
appropriate  statistic.  Also,  measures  which  examine  bias,  variability,  and 
skewness  are  presented. 

Still  the  analyst  must  choose  the  appropriate  summary  statistic.  But  unlike 
regression  analysis,  the  choice  is  explicit  and  can  be  consistently  applied  to  all 
candidate  estimating  procedures. 
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Statistics  Which  Depend  on  a  Particular  Estimating  Procedure 

A  final  set  of  statistics  can  be  calculated  from  the  historical  simulation  output 
by  making  use  of  statistical  model  assumptions  that  are  usually  associated  with 
the  particular  cost-estimating  procedure  under  examination.  An  example  is  the 
multiple  linear  regression  model,  which  is  usually  assumed  when  the  cost¬ 
estimating  procedure  of  interest  comprises  a  linear  EF  and  a  least-squares  technique. 

From  an  operational  point  of  view,  these  statistics  which  are  derived  from  an 
assumed  statistical  model  (applicable  to  an  EF)  are  not  very  versatile.  They  can 
only  be  used  to  compare  estimating  procedures  having  the  same  EF  and  they  also 
depend  on  the  validity  of  the  statistical  model  assumptions.  In  contrast,  the 
historical  simulation  data  summarizations  presented  in  the  previous  section  can 
therefore  be  used  to  compare  any  estimating  procedure. 

However,  the  estimating  procedure  specific  statistics  are  sometimes  worth 
examining.  Since  they  are  valid  for  any  estimating  procedures  which  utilize  the 
same  statistical  assumptions;  for  example,  the  class  of  linear  EFs  (with  least- 
squares  curve  fit),  they  can  be  used  to  compare  candidate  estimating  procedures 
in  the  class. ’  To  date  the  distribution  of  the  historical  simulation  output— the 
predictions  and  residuals— has  been  determined  for  the  usual  linear  regression 
assumptions.  Also,  a  goodness-of-fit  test  and  a  test  to  determine  if  there  is  bias 
present  has  been  defined  for  a  subset  of  historical  simulation  output.  These  are 
documented  in  references  cited  in  footnote  1. 

Summary  of  Historical  Simulation  Features 

Several  unique  advantages  of  the  historical  simulation  procedure  have  been 
identified.  These  are  summarized  below: 

—Historical  simulation  can  compare  a  wider  class  of  cost-estimating  procedures 
than  the  usual  regression  techniques. 

—Historical  simulation  provides  an  easy-to-communicate  summary  statistic 
useful  for  describing  the  accuracy  of  a  prediction.  This  summary  statistic  is 
average  proportional  (or  absolute)  error  or  one  of  its  weighted  forms.  It  describes 
how  well  the  cost-estimating  procedure  would  have  predicted  if  it  had  been  used 
in  the  past  to  make  predictions  of  the  now  historical  data. 

—Historical  simulation  provides  a  view  independent  of  the  usual  regression 
theory  approach.  The  candidates  cost-estimating  procedure  is  judged  on  how 
well  it  predicts  the  data  from  prior  data  points  and  not  on  how  well  the  candidate 
cost-estimating  procedure  fits  the  data  as  a  result. 

Historical  simulation  can  be  used  to  evaluate  CERs  derived  from  stepwise 
multiple  regression  programs.  These  programs  choose  the  candidate  CER  which 
fits  the  data  best  (in  a  least-squares  sense).  Historical  simulation  does  not  depend 
on  this  choice  criterion  and  therefore  is  able  to  evaluate  the  CER  independently  of 
the  stepwise  regression  choice  criterion. 


5.  These  comparisons  can  also  be  made  with  the  usual  evaluation  procedures,  i.e..  the  usual 
regression  statistics  and.  hence,  historical  simulation  does  not  otter  a  comparison  that  cannot  other¬ 
wise  be  made. 
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Historical  simulation  gives  useful  information  about  extrapolations.  Due  to 
its  dependence  on  predicting  from  past  data,  historical  simulation  can 
demonstrate  the  estimating  procedure's  ability  to  handle  extrapolations  implicit 
in  the  data  base.  The  extrapolation  is  in  the  time  direction  if  the  data  is  ordered  on 
time.  However,  the  extrapolation  can  be  based  on  any  dimension  of  interest  as 
long  as  the  data  is  ordered  in  that  dimension  prior  to  the  historical  simulation 
analysis. 

Additional  properties  include; 

—Comparability  of  selection  criteria  across  all  candidate  estimating  procedures. 
--Additional  information  to  help  hypothesize  a  new  cost-estimating  procedure 
candidate. 

—Exposure  of  questionable  sample  point  s  which  do  not  fit  in  with  the  prior  data 
base  in  terms  of  information  content  for  parameter  estimation  and  in  terms  of 
simulated  predictions. 

—The  possibility  of  uncovering  errors  in  an  estimating  procedure's  formulation 
which  would  not  be  uncovered  by  the  usual  regression  statistics.  Figure  1. 

Users  of  historical  simulation  are  still  required  to  select  a  definition  of  ac¬ 
curacy;  however,  there  is  value  in  making  this  choice  explicit  (instead  of  implicit 
as  is  often  done  by  regression  theory  users). 

But  in  the  final  analysis,  historical  simulation  is  an  additional  tool  which  can 
give  an  independent  check  on  the  more  traditional  techniques.  As  such  it  should 
augment  and  not  replace  the  usual  techniques.  1 1 
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